PaperlayerView exact revision on arXiv

Source-linked AI summary

SPICE: Semantic Pseudo-labeling for Image Clustering

Chuang Niu, Hongming Shan, Ge Wang

arXiv:2103.09382v3cs.CVcs.AI

TL;DR

Existing deep clustering methods can inaccurately estimate instance similarity or semantic discrepancy. SPICE separates these roles into a feature model and clustering head, trains them in three annotation-free stages with two pseudo-labeling algorithms, and achieves roughly 10% improvement over existing methods across six datasets.

  • Problem

    Existing deep clustering methods inaccurately estimate either feature similarity or semantic discrepancy, while label features can lose instance-level information needed for similarity measurement.

    Source: §I · p. 1

  • Method

    SPICE divides clustering into a feature model and clustering head, using contrastive learning, prototype pseudo-labeling, and reliable pseudo-labeling in three annotation-free training stages.

    Source: §I · p. 2

  • Results

    SPICE delivers an average 10% performance boost in adjusted rand index, normalized mutual information, and clustering accuracy over competing methods on six public datasets.

    Source: §VI · p. 12

  • Takeaways & Limitations

    SPICE significantly reduces the gap between unsupervised and supervised classification, reaching a 2% accuracy difference on CIFAR-10.

    Source: §I · p. 2, C. Clustering Performance Comparison · p. 9

Abstract

from arXiv · show

The similarity among samples and the discrepancy between clusters are two crucial aspects of image clustering. However, current deep clustering methods suffer from the inaccurate estimation of either feature similarity or semantic discrepancy. In this paper, we present a Semantic Pseudo-labeling-based Image ClustEring (SPICE) framework, which divides the clustering network into a feature model for measuring the instance-level similarity and a clustering head for identifying the cluster-level discrepancy. We design two semantics-aware pseudo-labeling algorithms, prototype pseudo-labeling, and reliable pseudo-labeling, which enable accurate and reliable self-supervision over clustering. Without using any ground-truth label, we optimize the clustering network in three stages: 1) train the feature model through contrastive learning to measure the instance similarity, 2) train the clustering head with the prototype pseudo-labeling algorithm to identify cluster semantics, and 3) jointly train the feature model and clustering head with the reliable pseudo-labeling algorithm to improve the clustering performance. Extensive experimental results demonstrate that SPICE achieves significant improvements (~10%) over existing methods and establishes the new state-of-the-art clustering results on six image benchmark datasets in terms of three popular metrics. Importantly, SPICE significantly reduces the gap between unsupervised and fully-supervised classification; e.g., there is only a 2% (91.8% vs 93.8%) accuracy difference on CIFAR-10. Our code has been made publically available at https://github.com/niuchuangnn/SPICE.

I. INTRODUCTION

Image clustering must estimate both instance similarity and semantic differences between clusters, yet existing methods often misestimate one of these aspects. SPICE addresses this with a divided clustering network and semantics-aware pseudo-labeling.

  • Image clustering groups images into meaningful clusters without human annotations while relying on sample similarity and semantic cluster discrepancy.Source: §I · p. 1
  • Existing methods can lose instance-level information when using category-level label features, limiting their ability to measure similarity accurately.Source: §I · p. 1
  • SPICE divides the clustering network into a feature model and clustering head, then trains them separately and jointly without annotations.Source: §I · p. 2
  • Prototype pseudo-labeling identifies cluster prototypes and assigns prototype labels to neighboring samples, reducing inconsistency among borderline samples.Source: §I · p. 2
  • Reliable pseudo-labeling selects reliable samples for joint training of the feature model and clustering head, improving clustering performance.Source: §I · p. 2

B. Unsupervised Representation Learning

SPICE treats unsupervised clustering as a classification-like task without annotations, using a feature model for representation learning and a clustering head for class probabilities. Its three-stage training progressively initializes and jointly refines both components.

  • Unsupervised representation learning maps images to semantically meaningful features without human annotations and supports downstream tasks such as detection and classification.Source: B. Unsupervised Representation Learning · p. 3
  • SPICE targets clustering with the exact number of real classes while training both the feature model and clustering head without annotations.Source: B. Unsupervised Representation Learning · p. 3
  • SPICE reframes the joint-training stage as a semi-supervised learning paradigm while using no annotations.Source: C. Semi-Supervised Classification · pp. 3–4
  • The clustering network maps each image to a feature vector and then to probabilities over K classes through its feature model and clustering head.Source: §III · p. 4
  • Training proceeds by contrastively optimizing the feature model, prototype-pseudo-labeling the clustering head with the feature model frozen, and then jointly optimizing both components.Source: §III · p. 4

A. Feature Model Training with Contrastive Learning

SPICE trains its feature model with instance-discrimination contrastive learning so transformed views of the same image are similar and different images remain discriminative. The resulting model can be used with other unsupervised representation-learning methods as well.

B. Clustering Head Training with Prototype Pseudo-Labeling

SPICE trains a clustering head with prototype pseudo-labels in an EM framework, using confident samples to estimate cluster prototypes and label their nearest neighbors.

Prototype pseudo-labeling

  • The feature model produces embedding features, while the clustering head maps them to probabilities over K clusters.The feature model is fixed during this stage, and the clustering head is a two-layer MLP.Source: B. Clustering Head Training with… · p. 5
  • The clustering head is trained separately from the feature model by alternating pseudo-label estimation and parameter optimization in an EM framework.The E-step estimates cluster labels, while the M-step optimizes the clustering-head parameters given those labels.Source: B. Clustering Head Training with… · p. 5
  • Top-confidence samples for each cluster estimate feature-space prototypes, whose nearest samples receive the corresponding prototype labels.Prototype pseudo-labeling uses predicted probabilities for selection and cosine similarity for nearest-neighbor assignment.Source: B. Clustering Head Training with… · p. 5
  • In a toy example, 3 confident samples per cluster estimate prototypes, followed by selecting 3 nearest samples for each prototype.Unselected samples receive -1 and are excluded from training.Source: B. Clustering Head Training with… · p. 5, Figure 4 · p. 6
  • Overlap assignment performs better than non-overlap assignment because non-overlap labeling may introduce local inconsistency for samples far from cluster centers.Overlap assignment allows one sample to receive labels from multiple clusters.Source: B. Clustering Head Training with… · p. 6
  • The clustering head is optimized with cross-entropy on strongly transformed images, while double softmax reduces learning speed for low-probability predictions.The feature model remains fixed, enabling multiple clustering heads to be trained independently and the best head to be selected by minimum loss.Source: B. Clustering Head Training with… · p. 6

C. Joint Training with Reliable Pseudo-Labeling

SPICE selects reliable pseudo-labels from neighborhood semantic consistency and jointly trains the feature model and clustering head with reliable-label and consistency objectives.

Reliable pseudo-labeling

  • Separate training can be suboptimal because imperfect features and clustering predictions may respectively mismatch semantic clusters or assign identical labels to dissimilar samples.These errors motivate jointly refining both components.Source: C. Joint Training with Reliable… · p. 6
  • For each sample, SPICE computes the ratio of nearest neighbors sharing its pseudo-label to identify semantic consistency.Nearest neighbors are selected by cosine similarity between embedding features.Source: C. Joint Training with Reliable… · p. 6
  • Samples with consistency ratio r_i greater than threshold λ are retained as reliably labeled, while other pseudo-labels are ignored.The retained samples form the subset X_r used for joint training.Source: C. Joint Training with Reliable… · p. 6

Joint training

  • Joint training combines reliable pseudo-label supervision with consistency pseudo-labels for all samples whose confident predictions pass a threshold.Weak and strong transformations of images are constrained through the adapted semi-supervised learning procedure.Source: C. Joint Training with Reliable… · pp. 6–7
  • The reliable pseudo-labeling stage supports a general framework that gradually trains the feature model, clustering head, and full network end-to-end without annotations.The authors note that other semi-supervised algorithms can also use the generated reliable samples.Source: C. Joint Training with Reliable… · p. 7

A. Benchmark Datasets and Evaluation Metrics

SPICE is evaluated on six image-clustering datasets spanning varied image sizes, dataset scales, and cluster counts, using ARI, NMI, and ACC.

Datasets

Evaluation metrics

B. Implementation Details

The experiments use ResNet backbones, fixed architectural and training settings, strong augmentation, and multiple clustering heads with the best selected by loss.

Architecture

  • SPICE uses ResNet18 and ResNet34 backbones, with a D-D-K classification head, D = 512, and K set to the target dataset’s class count.SPICE without joint training is denoted SPICE_s.Source: B. Implementation Details · p. 7

Training setup

  • Representation learning uses MoCo-v2, while weak augmentation uses flip-and-shift and strong augmentation combines Cutout with four RandAugment transformations.The strong augmentation strategies follow SCAN’s setup.Source: B. Implementation Details · p. 7
  • SPICE trains 10 clustering heads and selects the head with minimum loss for the final clustering result.Reliable pseudo-labeling uses N_s = 100 and λ = 0.95, with FixMatch confidence threshold η = 0.95 and batch size M = 1,000.Source: B. Implementation Details · p. 7

Comparison protocol

  • Table II compares methods trained and tested on merged whole datasets under matched data conditions, with SPICE using no extra data or annotations.The best results are highlighted in bold.Source: Table II · p. 8

C. Clustering Performance Comparison

SPICE consistently outperforms competing clustering methods across whole-dataset and split train/test settings. Improvements span ACC, NMI, and ARI, while joint training provides additional gains but hierarchical datasets remain challenging.

Whole-dataset setting

  • 8.8%, 12.6%, and 14.4% improvements in ACC, NMI, and ARI, respectively, over CC on STL10 in the whole-dataset setting.SPICE also improves ACC, NMI, and ARI by about 10% on ImageNet-Dog-15, CIFAR-10, CIFAR-100-20, and Tiny-ImageNet-200.Source: C. Clustering Performance Comparison · p. 8
  • Joint training improves ACC over separate training on all datasets, including 8.8% on CIFAR-10 and 7.0% on CIFAR-100-20.Source: C. Clustering Performance Comparison · p. 8
  • Tiny-ImageNet-200 remains difficult because hierarchical classes prevent reliable pseudo-labeling, leaving large-scale hierarchical clustering an open problem.Source: C. Clustering Performance Comparison · p. 8

Split train/test setting

Detailed STL10 comparison

D. Semi-Supervised Classification Comparison

SPICE is comparable to or better than state-of-the-art semi-supervised methods while also improving representation quality when reliable pseudo-labels are sufficiently accurate. Visualizations and ablations connect these results to semantically meaningful clusters and design choices such as overlap assignment and separate training.

Semi-supervised comparison

  • SPICE is comparable to and sometimes better than state-of-the-art semi-supervised learning methods using limited labeled samples.The comparison uses 250 labeled CIFAR-10 samples and 1,000 labeled STL10 samples.Source: D. Semi-Supervised Classification… · p. 9

Representation learning

  • 97.7% and 96.5% ACC for reliable labels on STL10 and CIFAR-10 support feature improvement after joint training.CIFAR-100-20 reliable labels reach 67.7% ACC, where feature improvement is not observed.Source: E. Unsupervised Representation Learning · p. 9

Visualization of cluster semantics

Ablation study

  • Overlap assignment is preferred because non-overlap assignment may introduce local inconsistency for samples far from cluster centers.Source: 2) Ablation study · p. 11
  • Freezing the feature model during clustering avoids degradation from inaccurate initial labels and the resulting harmful pseudo-labeling cycle.Source: 2) Ablation study · p. 11
  • Entropy loss is unnecessary for preventing trivial solutions with balanced assignment but becomes necessary to avoid empty clusters among 200 hierarchical Tiny-ImageNet classes.Source: 2) Ablation study · p. 11
  • Temperature cross-entropy outperforms plain cross-entropy, but remains inferior to the double-softmax formulation.The best tested temperature for TCE was 0.2.Source: 2) Ablation study · p. 11

3) Clustering head selection:

SPICE selects clustering heads without ground-truth labels by using classification loss as a proxy for clustering performance, then improves training through reliable pseudo-labels and augmentation choices.

3) Clustering head selection:

  • The classification loss on the whole test dataset approximates clustering performance, enabling head selection without ground-truth supervision.The selected head performs very close to the ground-truth-selected head, filtering out poorly performing heads.Source: 3) Clustering head selection · p. 11, Figure 6 · p. 11

Training stability and efficiency

  • SPICE has lower standard deviation than competing methods, while lightweight clustering heads can be trained independently and simultaneously without extra training time.This reduces variability across unsupervised trials while preserving efficient head training.Source: 3) Clustering head selection · p. 11

Reliable pseudo-labeling

  • 0.959 ACC for selected reliable labels and 0.926 ACC after joint training with reliable samples, versus 0.838 when using all pseudo-labeled samples.Reliable-label selection therefore provides cleaner supervision than using all pseudo-labeled samples in the illustrated CIFAR-10 experiment.Source: Figure 7 · p. 11, 4) Effect of reliable labeling · pp. 11–12

Data augmentation

  • Weak augmentation in the second branch and strong augmentation in the third branch produced the best performance on the whole STL10 dataset.Using strong augmentation in the second branch during labeling resulted in relatively worse performance.Source: 5) Effect of data augmentation · p. 12, Table VIII · p. 12

Limitations

  • Existing deep clustering methods assume that the number of clusters is known and that clusters contain similar numbers of samples.The paper identifies automatic cluster-number determination and handling nonuniform cluster sizes as open issues for real-world applications.Source: §V · p. 12
  • The three-stage progressive training process combines multiple algorithmic ingredients and could be unified into a framework that also optimizes weak and strong transforms.This refinement is stated to be beyond the scope of the paper.Source: §V · p. 12

VI. CONCLUSION

SPICE addresses image clustering by separately and jointly training a feature model and clustering head with semantic pseudo-labeling. Its double-softmax mechanism is designed for imperfect pseudo-labels, while experiments show strong performance across datasets.

VI. CONCLUSION

  • SPICE divides the clustering network into a feature model and clustering head, trained separately and then jointly with prototype and reliable pseudo-labeling.The three stages target instance similarity, cluster discrepancy, and joint improvement without annotations.Source: §VI · p. 12
  • 10% average performance boost on six public datasets across adjusted rand index, normalized mutual information, and clustering accuracy.SPICE is reported as comparable to or better than state-of-the-art semi-supervised learning methods and reduces the CIFAR-10 unsupervised-versus-supervised gap to 2%.Source: §VI · p. 12

Double softmax

  • A double-softmax mechanism reduces learning speed for low-probability predictions when training the clustering head with imperfect pseudo-labels.It also supports sample changes between clusters by relatively increasing learning speed for false high-probability predictions.Source: PSEUDO-LABELING-BASED IMAGE CLUSTERING · pp. 12–13
  • The double-softmax gradient is analyzed by comparing derivatives for single-softmax and double-softmax compositions under cross-entropy loss.The formulation uses K as the number of clusters and y_k as the pseudo label.Source: PSEUDO-LABELING-BASED IMAGE CLUSTERING · p. 13
  • With K = 8 clusters and pseudo-label c = 2, the simulation shows larger predicted probabilities generally produce larger gradient-ratio responses, with a notable false high-probability case at k = 5.The simulation is reported as consistent with the proposed gradient analysis.Source: PSEUDO-LABELING-BASED IMAGE CLUSTERING · p. 13

Related papers

Loading 2103.09382v3…