PaperlayerView exact revision on arXiv

Source-linked AI summary

SPICE: Semantic Pseudo-labeling for Image Clustering

Chuang Niu, Hongming Shan, Ge Wang

arXiv:2103.09382v3cs.CVcs.AI

TL;DR

Existing deep clustering methods can inaccurately estimate instance similarity or semantic discrepancy. SPICE separates these roles into a feature model and clustering head, trains them in three annotation-free stages with two pseudo-labeling algorithms, and achieves roughly 10% improvement over existing methods across six datasets.

  • Problem

    Existing deep clustering methods inaccurately estimate either feature similarity or semantic discrepancy, while label features can lose instance-level information needed for similarity measurement.

  • Method

    SPICE divides clustering into a feature model and clustering head, using contrastive learning, prototype pseudo-labeling, and reliable pseudo-labeling in three annotation-free training stages.

  • Results

    SPICE delivers an average 10% performance boost in adjusted rand index, normalized mutual information, and clustering accuracy over competing methods on six public datasets.

  • Takeaways & Limitations

    SPICE significantly reduces the gap between unsupervised and supervised classification, reaching a 2% accuracy difference on CIFAR-10.

Abstract

from arXiv · show

The similarity among samples and the discrepancy between clusters are two crucial aspects of image clustering. However, current deep clustering methods suffer from the inaccurate estimation of either feature similarity or semantic discrepancy. In this paper, we present a Semantic Pseudo-labeling-based Image ClustEring (SPICE) framework, which divides the clustering network into a feature model for measuring the instance-level similarity and a clustering head for identifying the cluster-level discrepancy. We design two semantics-aware pseudo-labeling algorithms, prototype pseudo-labeling, and reliable pseudo-labeling, which enable accurate and reliable self-supervision over clustering. Without using any ground-truth label, we optimize the clustering network in three stages: 1) train the feature model through contrastive learning to measure the instance similarity, 2) train the clustering head with the prototype pseudo-labeling algorithm to identify cluster semantics, and 3) jointly train the feature model and clustering head with the reliable pseudo-labeling algorithm to improve the clustering performance. Extensive experimental results demonstrate that SPICE achieves significant improvements (~10%) over existing methods and establishes the new state-of-the-art clustering results on six image benchmark datasets in terms of three popular metrics. Importantly, SPICE significantly reduces the gap between unsupervised and fully-supervised classification; e.g., there is only a 2% (91.8% vs 93.8%) accuracy difference on CIFAR-10. Our code has been made publically available at https://github.com/niuchuangnn/SPICE.

I. INTRODUCTION

Image clustering must estimate both instance similarity and semantic differences between clusters, yet existing methods often misestimate one of these aspects. SPICE addresses this with a divided clustering network and semantics-aware pseudo-labeling.

  • Image clustering groups images into meaningful clusters without human annotations while relying on sample similarity and semantic cluster discrepancy.
  • Existing methods can lose instance-level information when using category-level label features, limiting their ability to measure similarity accurately.
  • SPICE divides the clustering network into a feature model and clustering head, then trains them separately and jointly without annotations.
  • Prototype pseudo-labeling identifies cluster prototypes and assigns prototype labels to neighboring samples, reducing inconsistency among borderline samples.
  • Reliable pseudo-labeling selects reliable samples for joint training of the feature model and clustering head, improving clustering performance.

B. Unsupervised Representation Learning

SPICE treats unsupervised clustering as a classification-like task without annotations, using a feature model for representation learning and a clustering head for class probabilities. Its three-stage training progressively initializes and jointly refines both components.

  • Unsupervised representation learning maps images to semantically meaningful features without human annotations and supports downstream tasks such as detection and classification.
  • SPICE targets clustering with the exact number of real classes while training both the feature model and clustering head without annotations.
  • SPICE reframes the joint-training stage as a semi-supervised learning paradigm while using no annotations.
  • The clustering network maps each image to a feature vector and then to probabilities over K classes through its feature model and clustering head.
  • Training proceeds by contrastively optimizing the feature model, prototype-pseudo-labeling the clustering head with the feature model frozen, and then jointly optimizing both components.

A. Feature Model Training with Contrastive Learning

SPICE trains its feature model with instance-discrimination contrastive learning so transformed views of the same image are similar and different images remain discriminative. The resulting model can be used with other unsupervised representation-learning methods as well.

  • The feature model is trained with instance-discrimination-based unsupervised representation learning to measure similarity between image instances.
  • Two randomly transformed views of each image enter separate branches containing a feature model and projection head during contrastive training.
  • A memory bank maintains a queue of encoded negative samples to reduce the GPU-memory requirement of large-batch contrastive learning.
  • The loss encourages features from different transformations of one image to be similar while making features from different images discriminative.
  • The upper branch is updated as a moving average of the lower branch, with τ = 0.2 and µ = 0.999 used as hyperparameters.
  • SPICE can use any unsupervised representation-learning method and network architecture as its feature model.

B. Clustering Head Training with Prototype Pseudo-Labeling

SPICE trains a clustering head with prototype pseudo-labels in an EM framework, using confident samples to estimate cluster prototypes and label their nearest neighbors.

  • Prototype pseudo-labeling: The feature model produces embedding features, while the clustering head maps them to probabilities over K clusters.The feature model is fixed during this stage, and the clustering head is a two-layer MLP.
  • Prototype pseudo-labeling: The clustering head is trained separately from the feature model by alternating pseudo-label estimation and parameter optimization in an EM framework.The E-step estimates cluster labels, while the M-step optimizes the clustering-head parameters given those labels.
  • Prototype pseudo-labeling: Top-confidence samples for each cluster estimate feature-space prototypes, whose nearest samples receive the corresponding prototype labels.Prototype pseudo-labeling uses predicted probabilities for selection and cosine similarity for nearest-neighbor assignment.
  • Prototype pseudo-labeling: In a toy example, 3 confident samples per cluster estimate prototypes, followed by selecting 3 nearest samples for each prototype.Unselected samples receive -1 and are excluded from training.
  • Prototype pseudo-labeling: Overlap assignment performs better than non-overlap assignment because non-overlap labeling may introduce local inconsistency for samples far from cluster centers.Overlap assignment allows one sample to receive labels from multiple clusters.
  • Prototype pseudo-labeling: The clustering head is optimized with cross-entropy on strongly transformed images, while double softmax reduces learning speed for low-probability predictions.The feature model remains fixed, enabling multiple clustering heads to be trained independently and the best head to be selected by minimum loss.

C. Joint Training with Reliable Pseudo-Labeling

SPICE selects reliable pseudo-labels from neighborhood semantic consistency and jointly trains the feature model and clustering head with reliable-label and consistency objectives.

  • Reliable pseudo-labeling: Separate training can be suboptimal because imperfect features and clustering predictions may respectively mismatch semantic clusters or assign identical labels to dissimilar samples.These errors motivate jointly refining both components.
  • Reliable pseudo-labeling: For each sample, SPICE computes the ratio of nearest neighbors sharing its pseudo-label to identify semantic consistency.Nearest neighbors are selected by cosine similarity between embedding features.
  • Reliable pseudo-labeling: Samples with consistency ratio r_i greater than threshold λ are retained as reliably labeled, while other pseudo-labels are ignored.The retained samples form the subset X_r used for joint training.
  • Joint training: Joint training combines reliable pseudo-label supervision with consistency pseudo-labels for all samples whose confident predictions pass a threshold.Weak and strong transformations of images are constrained through the adapted semi-supervised learning procedure.
  • Joint training: The reliable pseudo-labeling stage supports a general framework that gradually trains the feature model, clustering head, and full network end-to-end without annotations.The authors note that other semi-supervised algorithms can also use the generated reliable samples.

A. Benchmark Datasets and Evaluation Metrics

SPICE is evaluated on six image-clustering datasets spanning varied image sizes, dataset scales, and cluster counts, using ARI, NMI, and ACC.

  • Datasets: The evaluation covers STL10, CIFAR10, CIFAR-100-20, ImageNet-10, ImageNet-Dog, and Tiny-ImageNet.These datasets reflect diversity in image sizes, numbers of images, and numbers of clusters.
  • Datasets: The selected datasets differ in image size, image count, and cluster count.Dataset specifications and partitions are summarized in Table I.
  • Evaluation metrics: Clustering performance is evaluated with Adjusted Rand Index, Normalized Mutual Information, and clustering Accuracy.These are the three popular metrics used in the experiments.

B. Implementation Details

The experiments use ResNet backbones, fixed architectural and training settings, strong augmentation, and multiple clustering heads with the best selected by loss.

  • Architecture: SPICE uses ResNet18 and ResNet34 backbones, with a D-D-K classification head, D = 512, and K set to the target dataset’s class count.SPICE without joint training is denoted SPICE_s.
  • Training setup: Representation learning uses MoCo-v2, while weak augmentation uses flip-and-shift and strong augmentation combines Cutout with four RandAugment transformations.The strong augmentation strategies follow SCAN’s setup.
  • Training setup: SPICE trains 10 clustering heads and selects the head with minimum loss for the final clustering result.Reliable pseudo-labeling uses N_s = 100 and λ = 0.95, with FixMatch confidence threshold η = 0.95 and batch size M = 1,000.
  • Comparison protocol: Table II compares methods trained and tested on merged whole datasets under matched data conditions, with SPICE using no extra data or annotations.The best results are highlighted in bold.

C. Clustering Performance Comparison

SPICE consistently outperforms competing clustering methods across whole-dataset and split train/test settings. Improvements span ACC, NMI, and ARI, while joint training provides additional gains but hierarchical datasets remain challenging.

  • Whole-dataset setting: 8.8%, 12.6%, and 14.4% improvements in ACC, NMI, and ARI, respectively, over CC on STL10 in the whole-dataset setting.SPICE also improves ACC, NMI, and ARI by about 10% on ImageNet-Dog-15, CIFAR-10, CIFAR-100-20, and Tiny-ImageNet-200.
  • Whole-dataset setting: Joint training improves ACC over separate training on all datasets, including 8.8% on CIFAR-10 and 7.0% on CIFAR-100-20.
  • Whole-dataset setting: Tiny-ImageNet-200 remains difficult because hierarchical classes prevent reliable pseudo-labeling, leaving large-scale hierarchical clustering an open problem.
  • Split train/test setting: 6.5%, 9.4%, and 11.5% gains in ACC, NMI, and ARI on STL10 over SCANMoCo under split-dataset evaluation.SPICE also improves ACC, NMI, and ARI by 4.4%, 6.4%, and 8.0% on CIFAR-10, and by 8.0%, 9.3%, and 9.4% on CIFAR-100-20.
  • Split train/test setting: SPICE significantly reduces the supervised classification gap, reaching only a 2% ACC difference on CIFAR-10.
  • Detailed STL10 comparison: SPICE outperforms MoCo+k-means and SCAN*MoCo in both mean performance and standard deviation, and also exceeds SCANMoCo in mean performance and stability.

D. Semi-Supervised Classification Comparison

SPICE is comparable to or better than state-of-the-art semi-supervised methods while also improving representation quality when reliable pseudo-labels are sufficiently accurate. Visualizations and ablations connect these results to semantically meaningful clusters and design choices such as overlap assignment and separate training.

  • Semi-supervised comparison: SPICE is comparable to and sometimes better than state-of-the-art semi-supervised learning methods using limited labeled samples.The comparison uses 250 labeled CIFAR-10 samples and 1,000 labeled STL10 samples.
  • Representation learning: 97.7% and 96.5% ACC for reliable labels on STL10 and CIFAR-10 support feature improvement after joint training.CIFAR-100-20 reliable labels reach 67.7% ACC, where feature improvement is not observed.
  • Visualization of cluster semantics: Prototype visualizations match human annotations, while attention maps focus on semantic objects such as dogs across different image locations.
  • Ablation study: Overlap assignment is preferred because non-overlap assignment may introduce local inconsistency for samples far from cluster centers.
  • Ablation study: Freezing the feature model during clustering avoids degradation from inaccurate initial labels and the resulting harmful pseudo-labeling cycle.
  • Ablation study: Entropy loss is unnecessary for preventing trivial solutions with balanced assignment but becomes necessary to avoid empty clusters among 200 hierarchical Tiny-ImageNet classes.
  • Ablation study: Temperature cross-entropy outperforms plain cross-entropy, but remains inferior to the double-softmax formulation.The best tested temperature for TCE was 0.2.

3) Clustering head selection:

SPICE selects clustering heads without ground-truth labels by using classification loss as a proxy for clustering performance, then improves training through reliable pseudo-labels and augmentation choices.

  • 3) Clustering head selection:: The classification loss on the whole test dataset approximates clustering performance, enabling head selection without ground-truth supervision.The selected head performs very close to the ground-truth-selected head, filtering out poorly performing heads.
  • Training stability and efficiency: SPICE has lower standard deviation than competing methods, while lightweight clustering heads can be trained independently and simultaneously without extra training time.This reduces variability across unsupervised trials while preserving efficient head training.
  • Reliable pseudo-labeling: 0.959 ACC for selected reliable labels and 0.926 ACC after joint training with reliable samples, versus 0.838 when using all pseudo-labeled samples.Reliable-label selection therefore provides cleaner supervision than using all pseudo-labeled samples in the illustrated CIFAR-10 experiment.
  • Data augmentation: Weak augmentation in the second branch and strong augmentation in the third branch produced the best performance on the whole STL10 dataset.Using strong augmentation in the second branch during labeling resulted in relatively worse performance.
  • Limitations: Existing deep clustering methods assume that the number of clusters is known and that clusters contain similar numbers of samples.The paper identifies automatic cluster-number determination and handling nonuniform cluster sizes as open issues for real-world applications.
  • Limitations: The three-stage progressive training process combines multiple algorithmic ingredients and could be unified into a framework that also optimizes weak and strong transforms.This refinement is stated to be beyond the scope of the paper.

VI. CONCLUSION

SPICE addresses image clustering by separately and jointly training a feature model and clustering head with semantic pseudo-labeling. Its double-softmax mechanism is designed for imperfect pseudo-labels, while experiments show strong performance across datasets.

  • VI. CONCLUSION: SPICE divides the clustering network into a feature model and clustering head, trained separately and then jointly with prototype and reliable pseudo-labeling.The three stages target instance similarity, cluster discrepancy, and joint improvement without annotations.
  • VI. CONCLUSION: 10% average performance boost on six public datasets across adjusted rand index, normalized mutual information, and clustering accuracy.SPICE is reported as comparable to or better than state-of-the-art semi-supervised learning methods and reduces the CIFAR-10 unsupervised-versus-supervised gap to 2%.
  • Double softmax: A double-softmax mechanism reduces learning speed for low-probability predictions when training the clustering head with imperfect pseudo-labels.It also supports sample changes between clusters by relatively increasing learning speed for false high-probability predictions.
  • Double softmax: The double-softmax gradient is analyzed by comparing derivatives for single-softmax and double-softmax compositions under cross-entropy loss.The formulation uses K as the number of clusters and y_k as the pseudo label.
  • Double softmax: With K = 8 clusters and pseudo-label c = 2, the simulation shows larger predicted probabilities generally produce larger gradient-ratio responses, with a notable false high-probability case at k = 5.The simulation is reported as consistent with the proposed gradient analysis.
Loading 2103.09382v3…