PaperlayerView exact revision on arXiv

Source-linked AI summary

SCAN: Learning to Classify Images without Labels

Wouter Van Gansbeke, Simon Vandenhende, Stamatios Georgoulis, Marc Proesmans, Luc Van Gool

arXiv:2005.12320v2cs.CVcs.LG

TL;DR

Unsupervised image classification asks whether images can be grouped into semantically meaningful clusters without ground-truth annotations. SCAN decouples feature learning and clustering: self-supervised representation learning supplies features, which become a prior for learnable clustering. The method outperforms prior work by large margins across multiple datasets and reports positive results on ImageNet.

  • Problem

    Unsupervised image classification seeks semantically meaningful image clusters when ground-truth labels are absent, and the semantic classes or their number may be unknown.

  • Method

    SCAN first learns image representations through a self-supervised pretext task, then uses feature-similarity nearest neighbors as a prior for learnable semantic clustering.

  • Results

    SCAN outperforms prior work by large margins across multiple datasets and reports promising results on the large-scale ImageNet dataset.

  • Takeaways & Limitations

    The results support applying semantic clustering to large-scale datasets without ground-truth annotations.

  • Takeaways & Limitations

    Overclustering comparisons use a many-to-one mapping and therefore are not entirely fair, serving only as an indication.

Abstract

from arXiv · show

Can we automatically group images into semantically meaningful clusters when ground-truth annotations are absent? The task of unsupervised image classification remains an important, and open challenge in computer vision. Several recent approaches have tried to tackle this problem in an end-to-end fashion. In this paper, we deviate from recent works, and advocate a two-step approach where feature learning and clustering are decoupled. First, a self-supervised task from representation learning is employed to obtain semantically meaningful features. Second, we use the obtained features as a prior in a learnable clustering approach. In doing so, we remove the ability for cluster learning to depend on low-level features, which is present in current end-to-end learning approaches. Experimental evaluation shows that we outperform state-of-the-art methods by large margins, in particular +26.6% on CIFAR10, +25.0% on CIFAR100-20 and +21.3% on STL10 in terms of classification accuracy. Furthermore, our method is the first to perform well on a large-scale dataset for image classification. In particular, we obtain promising results on ImageNet, and outperform several semi-supervised learning methods in the low-data regime without the use of any ground-truth annotations. The code is made publicly available at https://github.com/wvangansbeke/Unsupervised-Classification.

1 Introduction and prior work

Unsupervised image classification seeks semantic clusters without ground-truth labels, but existing approaches either use self-supervised features mainly for later supervised fine-tuning or jointly learn features and clusters. SCAN instead decouples feature learning from clustering in a two-step framework.

  • Problem: Unsupervised image classification groups images into semantically similar clusters without requiring predefined labels or even a known number of classes.Images within a cluster should share semantic classes, while images across clusters should be semantically dissimilar.
  • Prior work: Self-supervised representation learning obtains visual features directly from images through annotation-free pretext tasks, but is mainly used before supervised fine-tuning.Examples include predicting patch context, solving jigsaw puzzles, colorizing images, predicting rotations, and instance discrimination.
  • Prior work: End-to-end methods combine feature learning and clustering, refining clusters from confident samples, offline reassignments, or learned clustering functions.These approaches use the CNN architecture or clustering objectives as priors during joint learning.
  • SCAN: SCAN uses a two-step approach that first learns feature representations through a pretext task and then clusters images using those representations as a prior.This separates semantic feature learning from cluster learning, unlike recent end-to-end approaches.
  • SCAN: Nearest neighbors mined by feature similarity provide the clustering prior because they usually belong to the same semantic class.SCAN classifies each image together with its mined neighbors using a loss that encourages consistent and discriminative predictions.
  • Results: SCAN outperforms prior work by large margins across multiple datasets and reports promising results on large-scale ImageNet.The authors present this as evidence favoring separation between learning semantically meaningful features and clustering them.

2 Method

SCAN separates representation learning from clustering: it mines semantically meaningful nearest neighbors, then jointly clusters each image with those neighbors while refining predictions through self-labeling.

  • 2.1 Representation learning for semantic clustering: SCAN first learns an embedding with a self-supervised pretext task, then mines each image’s K nearest neighbors as a clustering prior.The prior is derived from feature similarity rather than ground-truth annotations.
  • 2.1 Representation learning for semantic clustering: Nearest neighbors are used because they largely belong to the same semantic cluster across CIFAR10, CIFAR100-20, STL10, and ImageNet.The observation holds across different values of K.
  • 2.2 A semantic clustering loss: The clustering loss maximizes agreement between each image and its neighbors while an entropy term prevents all samples from collapsing into one cluster.The dot product is maximal for confident, one-hot predictions assigned to the same cluster.
  • 2.2 A semantic clustering loss: SCAN assumes the exact cluster count is unknown in practice, so it estimates the number of clusters, can overcluster, and enforces a uniform class distribution.For evaluation, the number of clusters is set equal to the number of ground-truth clusters.
  • 2 Method: Algorithm 1 implements SCAN as sequential pretext-task, clustering, and self-labeling steps, returning the dataset divided over C clusters.The clustering step uses the SCAN loss, while self-labeling uses cross-entropy on confident predictions.
  • 2.3 Fine-tuning through self-labeling: Self-labeling selects samples with pmax > threshold, assigns pseudo-labels from predicted clusters, and updates the network using cross-entropy on strongly augmented samples.As predictions become more certain, additional samples can enter the self-labeling process.

3 Experiments

Experiments evaluate SCAN across multiple datasets, ablations, robustness settings, and comparisons with prior work. The method achieves large gains, remains effective under overclustering, and produces semantically coherent clusters.

  • Datasets: The evaluation covers CIFAR10, CIFAR100-20, STL10, and ImageNet, using train and validation splits to study generalization to unseen examples.The experiments first focus on smaller datasets and discuss ImageNet separately.
  • Ablation studies: 87.6% versus 81.8%: self-labeling further improves CIFAR10 cluster-assignment accuracy.The network gradually becomes more confident during self-labeling, but a shift in augmentations is required for successful application.
  • Ablation studies: 83.5% and 87.6% for instance discrimination versus 74.3% for RotNet: augmentation-invariant pretext tasks produce better nearest neighbors.All evaluated pretext-task cases exceed 70% accuracy, but invariance between images and augmentations is associated with stronger results.
  • Ablation studies: 56.3% versus 79.3% on CIFAR10, 24.6% versus 41.1% on CIFAR100-20, and 47.70% versus 69.8% on STL10: using neighbors outperforms using only augmentations.Performance remains stable as K increases to 50 and improves up to a certain neighbor count.
  • Comparison with the state-of-the-art: +26.6% on CIFAR10, +25.0% on CIFAR100-20, and +21.3% on STL10: SCAN consistently outperforms prior work in clustering accuracy.The method also outperforms applying K-means to pretext features and obtains close to supervised performance on CIFAR10 and STL10.
  • Overclustering: Overestimating the cluster count preserves CIFAR10 accuracy at 87.6% to 86.2% and STL10 accuracy at 76.7% to 76.8%, while CIFAR100-20 improves from 45.9% to 55.1%.The authors conclude that exact knowledge of the number of clusters is not required, while noting that the overclustering comparison uses a many-to-one mapping.

4 Conclusion

The paper presents a framework for unsupervised image classification that outperforms prior work across varied datasets and produces positive ImageNet results. The authors identify semantic clustering as applicable to large-scale datasets and suggest extensions to other domains.

  • The proposed framework targets unsupervised image classification and offers advantages over recent end-to-end approaches.
  • The method outperforms prior work by large margins across a variety of datasets.
  • Positive ImageNet results demonstrate that semantic clustering can be applied to large-scale datasets.
  • The authors suggest extensions to semantic segmentation, semi-supervised learning, and few-shot learning.

A Smaller datasets

The supplementary material reports additional qualitative results on CIFAR10, CIFAR100-20, and STL10 using models from the state-of-the-art comparison.

  • Additional qualitative results are included for the smaller datasets.
  • The evaluated smaller datasets are CIFAR10, CIFAR100-20, and STL10.
  • The qualitative analysis uses models from the state-of-the-art comparison.

A.1 Prototypical examples

Prototype images generally make the clustered objects recognizable, while low-confidence predictions often involve visually difficult examples. Prototype alignment with ground-truth classes depends on the dataset’s class structure.

  • Prototypical examples: Prototype images are visualized for every cluster on CIFAR10, CIFAR100-20, and STL10.
  • Prototypical examples: The object of interest is clearly recognizable in the prototype images, with CIFAR10 and STL10 examples matching ground-truth classes.
  • Prototypical examples: CIFAR100-20 prototypes need not match individual ground-truth classes because the dataset uses superclasses.
  • Low confidence predictions: Low-confidence predictions are usually difficult to label and include partially visible, occluded, or poorly lit objects.

B.1 Training setup

ImageNet training uses instance discrimination with a ResNet-50 backbone, followed by frozen-backbone clustering and self-labeling fine-tuning. The setup specifies separate linear heads, augmentation strategies, and exponential moving-average updates.

  • Pretext Task: Instance discrimination is selected as the ImageNet pretext task with MoCo and a ResNet-50 backbone.
  • Clustering Step: During clustering, the backbone is frozen and only the final linear layer is trained with the SCAN loss.
  • Clustering Step: Ten linear heads are trained in parallel, after which the head with the lowest loss continues through self-labeling.
  • Self-Labeling Step: Self-labeling uses RandAugment, SGD with momentum 0.9, a constant learning rate of 0.03, and 25 epochs.
  • Self-Labeling Step: Model weights are updated through an exponential moving average with α = 0.999, without class balancing in the cross-entropy loss.

B.2 ImageNet - Subsets

On ImageNet subsets, the model’s errors concentrate on visually similar classes and challenging scenes, while cluster prototypes usually correspond to ground-truth classes. Full ImageNet qualitative results show remaining failures from background reliance and similar-looking images.

  • Most confusion occurs between visually difficult classes, such as black dog breeds or visually similar foods.Examples include Giant Schnauzer versus Flat-coated Retriever and Guacamole versus Mashed Potato.
  • The vast majority of ImageNet-50 cluster prototypes can be matched to ground-truth classes.
  • Low-confidence predictions often involve multiple visible objects, partial object visibility, distracting scene elements, or ambiguous objects of interest.
  • On full ImageNet, failure cases arise when the model overemphasizes backgrounds or cannot discriminate between similarly looking images.

C.1 Datasets

The evaluation uses train-validation splits to study generalization, with datasets and transformations documented across supplementary tables and figures. The supplement also records implementation updates, including revised ImageNet results after correcting a coding mistake.

  • Datasets: Train-validation splits replace full-dataset training and evaluation to study generalization properties and enable comparison with supervised methods.
  • Qualitative evaluation: Supplementary figures provide ImageNet-50 confusion, prototype, and low-confidence examples, plus ImageNet-1000 cluster examples and incorrect clusters.
  • Experimental setup: Strong augmentations combine four randomly selected RandAugment transformations followed by Cutout, using the same strategy across datasets.
  • Changes since version 1: The revised version updates experiments with SimCLR and MoCo implementations, train-validation evaluation, augmentation and threshold analyses, and corrected ImageNet results.
Loading 2005.12320v2…