PaperlayerView exact revision on arXiv

Source-linked AI summary

Contrastive Clustering

Yunfan Li, Peng Hu, Zitao Liu, Dezhong Peng, Joey Tianyi Zhou, Xi Peng

arXiv:2009.09687v1cs.LGcs.CVstat.ML

TL;DR

Deep clustering on complex datasets is limited by representation learning and by offline or alternating procedures. CC addresses this with one-stage, end-to-end dual contrastive learning over instance and cluster representations, outperforming competitive methods across six image benchmarks.

  • Problem

    Deep clustering seeks better representations for complex datasets while moving toward end-to-end learning instead of offline or alternating procedures.

  • Method

    CC jointly performs instance- and cluster-level contrastive learning in the row and column spaces of a feature matrix whose rows represent instance soft labels and columns represent clusters.

  • Results

    CC significantly outperforms state-of-the-art methods on six challenging image datasets, achieving up to 39% NMI improvement on CIFAR-100 over the most competitive baseline.

  • Takeaways & Limitations

    The unified dual-contrastive framework provides clustering-favorable representations while supporting one-stage, batch-wise optimization for large-scale online scenarios.

Abstract

from arXiv · show

In this paper, we propose a one-stage online clustering method called Contrastive Clustering (CC) which explicitly performs the instance- and cluster-level contrastive learning. To be specific, for a given dataset, the positive and negative instance pairs are constructed through data augmentations and then projected into a feature space. Therein, the instance- and cluster-level contrastive learning are respectively conducted in the row and column space by maximizing the similarities of positive pairs while minimizing those of negative ones. Our key observation is that the rows of the feature matrix could be regarded as soft labels of instances, and accordingly the columns could be further regarded as cluster representations. By simultaneously optimizing the instance- and cluster-level contrastive loss, the model jointly learns representations and cluster assignments in an end-to-end manner. Extensive experimental results show that CC remarkably outperforms 17 competitive clustering methods on six challenging image benchmarks. In particular, CC achieves an NMI of 0.705 (0.431) on the CIFAR-10 (CIFAR-100) dataset, which is an up to 19\% (39\%) performance improvement compared with the best baseline.

1 INTRODUCTION

Contrastive Clustering addresses weak representation learning and offline limitations in deep clustering by jointly learning instance representations and cluster assignments through dual contrastive learning. It operates in one-stage, end-to-end fashion and substantially outperforms state-of-the-art methods on six image datasets.

  • Motivation: Deep clustering uses neural networks to improve representations for complex datasets, but existing approaches include offline or alternating procedures.The introduction motivates end-to-end learning by contrasting representation limitations and accumulated errors in prior approaches.
  • Key observation: CC treats feature-matrix rows as instance soft labels and columns as cluster representations, unifying clustering with representation learning.The row-column interpretation enables clustering predictions to be recast as a special representation-learning task.
  • Method: CC constructs augmented data pairs and applies instance- and cluster-level contrastive learning in the feature matrix’s row and column spaces.Positive pairs are gathered and negative pairs scattered at both levels to learn discriminative features and clustering assignments.
  • Contributions: The dual contrastive framework is presented as clustering-specified learning that produces representations favorable for clustering.The authors distinguish CC from methods designed only to learn general, reusable representations.
  • Contributions: One-stage, end-to-end optimization requires only batch-wise updates, enabling application to large-scale online scenarios.The model jointly learns representations and clustering rather than alternating between separate stages.
  • Results: CC significantly outperforms state-of-the-art methods on all six datasets and achieves up to 39% NMI improvement on CIFAR-100 over the strongest baseline.The evaluated benchmarks include CIFAR-10/100, STL-10, ImageNet-10/Dogs, and Tiny-ImageNet.

2 RELATED WORK

The related work positions CC against contrastive-learning methods that operate at the instance level and deep-clustering methods that alternate representation learning with clustering. CC instead combines clustering-oriented contrastive learning with a unified row-column feature-matrix formulation.

  • Contrastive Learning: Contrastive learning maximizes similarities between positive pairs and minimizes similarities between negative pairs in a learned feature space.Under unsupervised learning, data augmentations can construct positive views of the same instance and define other pairs as negative.
  • Contrastive Learning: Existing contrastive-learning methods generally learn general-purpose representations through instance-level contrastive learning.CC differs by explicitly adding a cluster-level contrastive objective designed for clustering.
  • Deep Clustering: Traditional deep-clustering methods can jointly learn representations and assignments, but alternating procedures may accumulate errors.Examples include iterative agglomerative clustering and alternating k-means-based updates.
  • Deep Clustering: CC treats labels as special representations, conducting instance- and cluster-level representation learning in the feature matrix’s row and column spaces.This unified formulation differs from methods that mainly exploit neural networks’ representational capacity for clustering.
  • Deep Clustering: A clustering-oriented contrastive-learning paradigm is described as promoting clustering by minimizing inter-cluster similarities.The paper characterizes this as one of the first attempts to promote clustering through contrastive learning.

3 METHOD

Contrastive Clustering jointly learns instance representations and cluster assignments through two contrastive heads operating on feature-matrix rows and columns. It uses augmented data pairs, soft-label cluster outputs, and a one-stage objective combining instance- and cluster-level losses.

  • Pair Construction Backbone: CC constructs two augmented views for each input and extracts their features with a shared neural network backbone.The method uses stochastic transformations, including ResizedCrop, ColorJitter, Grayscale, HorizontalFlip, and GaussianBlur.
  • Instance-level Contrastive Head: The instance-level contrastive head applies contrastive learning to projected features, treating corresponding augmented views as positive pairs and other batch views as negatives.The loss is computed over every augmented sample and uses an instance-level temperature parameter.
  • Cluster-level Contrastive Head: The cluster-level head projects features into M dimensions, where rows represent sample soft labels and columns represent cluster representations.Each column pair from the two augmentations is positive, while the remaining 2M − 2 cluster pairs are negative.
  • Cluster-level Contrastive Head: An entropy term discourages assigning most instances to the same cluster by regularizing cluster assignment probabilities within each mini-batch.The assignment matrix has N rows and M columns, with each row interpreted as a sample’s cluster-probability distribution.
  • Objective Function: CC jointly optimizes the instance- and cluster-level contrastive losses in a one-stage, end-to-end process using their sum as the overall objective.After training, cluster assignments are obtained by selecting the maximum output of the cluster-level head.

4 EXPERIMENTS

This section evaluates the proposed Contrastive Clustering method through experiments designed to verify its effectiveness.

  • The experiments verify the effectiveness of the proposed Contrastive Clustering method.
  • The evaluation focuses on the proposed method rather than introducing a separate experimental objective.
  • The section frames experiments as evidence for the method’s effectiveness.

4.1 Experimental Configurations

The evaluation uses six image datasets, standard clustering metrics, and a common ResNet34-based implementation with specified optimization settings.

  • Datasets: The evaluation covers CIFAR-10/100, STL-10, ImageNet-10/Dogs, and Tiny-ImageNet.CIFAR-10, CIFAR-100, and STL-10 use training and test sets, whereas the other datasets use only training data.
  • Implementation Details: The implementation adopts ResNet34 as the backbone and resizes all inputs to accommodate its image-size design.
  • Implementation Details: The instance-level contrastive head uses a 128-dimensional row space and fixes its temperature parameter τI at 0.5.
  • Implementation Details: Adam jointly optimizes both contrastive heads and the backbone with learning rate 0.0003, batch size 256, and 1,000 training epochs.
  • Implementation Details: Training requires about 20–160 GPU-hours depending on the dataset.The reported costs are 70 GPU-hours for CIFAR-10, 90 for CIFAR-100, 160 for STL-10, 20 for ImageNet-10, 30 for ImageNet-Dogs, and 130 for Tiny-ImageNet.
  • Metrics: Three clustering metrics—NMI, ACC, and ARI—measure performance, with higher values indicating better clustering.

4.2 Comparisons with State of the Arts

Contrastive Clustering is compared with 17 representative clustering methods on six challenging image benchmarks and significantly outperforms the state-of-the-art baselines.

  • CC is compared with 17 representative state-of-the-art clustering approaches on six challenging image benchmarks.The comparison includes methods ranging from k-means and spectral clustering to deep clustering approaches such as IIC and PICA.
  • CC significantly outperforms the state-of-the-art baselines by a large margin on all six datasets.
  • 0.114, 0.121, and 0.153 are CC’s NMI margins over PICA on CIFAR-10, CIFAR-100, and STL-10, respectively.
  • More than 50% performance improvements over the best baseline are reported for CC on CIFAR-100 and Tiny-ImageNet in terms of ARI.

4.3 Qualitative Study

The qualitative study examines training dynamics on ImageNet-10 through pair-wise similarities and the evolution of instance features and cluster assignments.

  • Analysis Scope: The study analyzes pair-wise similarity changes and the evolution of learned representations and assignments during training on ImageNet-10.
  • Pair-wise Similarity: Positive instance and cluster similarities increase during training, while negative-pair similarities remain low.
  • Pair-wise Similarity: The positive–negative similarity interval is comparatively large at both instance and cluster levels.Negative instance variance is linked to same-class pairs, while positive cluster variance reflects inconsistent assignments across augmentations.
  • Feature and Assignment Evolution: At training onset, features are mixed and most instances occupy only a few clusters.
  • Feature and Assignment Evolution: As training proceeds, cluster assignments become more reasonable and features scatter and gather more distinctly.

4.4 Ablation Study

The ablations examine data augmentation, the two contrastive heads, and backbone dependence. Results indicate that augmentation and backbone representability affect clustering performance, while the two heads contribute jointly in some settings.

  • Importance of Data Augmentation: Data augmentation enhances CC performance, especially on the more complicated CIFAR-10 dataset.Removing augmentations causes each positive pair to consist of two same samples or clusters.
  • Effect of Contrastive Head: Removing either contrastive head changes the clustering setup and produces dataset-dependent effects.Without the cluster-level head, cluster assignments are obtained by applying k-means in instance space.
  • Reliance on Backbone Network: The ablation studies assess CC using three ResNet backbones of different depths.The study tests how much clustering performance depends on backbone structure.
  • Reliance on Backbone Network: Backbone representability contributes to clustering performance, while ResNet18 is sufficiently powerful on relatively simple ImageNet-10.The reported comparison uses ResNets with different depths.

5 CONCLUSION

The conclusion presents Contrastive Clustering as a unified method that performs instance- and cluster-level contrastive learning. It reports promising clustering performance and identifies semi-supervised and transfer learning as future directions.

  • Conclusion: CC represents instances and clusters through the rows and columns of a feature matrix, respectively.This observation motivates the method's unified representation-learning framework.
  • Conclusion: CC conducts contrastive learning at the instance and cluster levels under one unified framework.The method is designed for clustering.
  • Conclusion: The paper reports promising clustering performance for the proposed CC method.The conclusion states this result without specifying a particular benchmark value.
  • Conclusion: Future work will extend CC to applications such as semi-supervised learning and transfer learning.These are proposed directions rather than reported results.
Loading 2009.09687v1…