Source-linked AI summary
Graph Contrastive Clustering
Huasong Zhong, Jianlong Wu, Chong Chen, Jianqiang Huang, Minghua Deng, Liqiang Nie, Zhouchen Lin, Xian-Sheng Hua
TL;DR
Existing contrastive clustering methods do not fully incorporate category information or the clustering objective, limiting representation suitability for clustering. GCC introduces graph contrastive learning at both representation and assignment levels, extending consistency from instances to clusters. Across six benchmarks, the paper reports that GCC learns more clustering-friendly representations and outperforms a wide range of state-of-the-art methods.
Problem
Existing contrastive clustering methods do not incorporate category information and the clustering objective sufficiently, so their learned representations may not be optimal for clustering.
Method
GCC applies graph contrastive learning to representation and assignment learning, using graph-based objectives and cluster-level consistency.
Results
Across six deep clustering benchmarks, GCC learns more clustering-friendly representations and outperforms a wide range of state-of-the-art methods.
Takeaways & Limitations
Incorporating latent category information at instance and cluster levels supports more discriminative features and compact clustering assignments.
Takeaways & Limitations
GCC mainly learns object shape, so visually similar patterns across classes may be grouped together and varied patterns within a class may be separated.
Abstract
from arXiv · showhide
Recently, some contrastive learning methods have been proposed to simultaneously learn representations and clustering assignments, achieving significant improvements. However, these methods do not take the category information and clustering objective into consideration, thus the learned representations are not optimal for clustering and the performance might be limited. Towards this issue, we first propose a novel graph contrastive learning framework, which is then applied to the clustering task and we come up with the Graph Constrastive Clustering~(GCC) method. Different from basic contrastive clustering that only assumes an image and its augmentation should share similar representation and clustering assignments, we lift the instance-level consistency to the cluster-level consistency with the assumption that samples in one cluster and their augmentations should all be similar. Specifically, on the one hand, the graph Laplacian based contrastive loss is proposed to learn more discriminative and clustering-friendly features. On the other hand, a novel graph-based contrastive learning strategy is proposed to learn more compact clustering assignments. Both of them incorporate the latent category information to reduce the intra-cluster variance while increasing the inter-cluster variance. Experiments on six commonly used datasets demonstrate the superiority of our proposed approach over the state-of-the-art methods.
1. Introduction
Clustering is difficult without labels because same-class samples vary in appearance, while existing methods may learn indiscriminative features or separate representation learning from clustering. GCC addresses these limitations by applying graph contrastive learning at both instance and cluster levels.
- Unlabeled clustering is challenging because samples from the same class can have varied appearances and supervision is unavailable.
- Classic clustering methods suffer from indiscriminative representations and sub-optimal clustering caused by separating feature extraction from clustering.
- GCC assumes samples within one cluster and their augmentations should share similar representations and assignments, extending consistency from instances to clusters.
- GCC uses a graph Laplacian contrastive loss for clustering-friendly features and graph-based contrastive learning for compact assignments.
- Experiments on six challenging datasets report significant improvements, with ablations assessing the effectiveness of each proposed module.
2. Related work
Deep clustering methods include reconstruction-based and self-augmentation-based approaches, but existing methods can overlook the connection between representation learning and cluster assignment learning.
- Deep clustering methods are mainly divided into reconstruction-based and self-augmentation-based categories.
- Reconstruction-based methods use auto-encoders and impose regularization terms on latent feature learning.
- Self-augmentation methods include approaches that optimize reconstruction, partition confidence, invariant features, or multistage clustering objectives.
- Existing approaches achieve good results but ignore connections between cluster assignment learning and representation learning, whereas GCC considers them jointly.
3.1. Problem Formulation
Deep clustering maps unlabeled images into normalized representation features and probability-based cluster assignments so semantically similar images can be grouped together.
- Deep clustering aims to separate N unlabeled images into K clusters using a convolutional neural network.
- The learned mapping assigns each image a d-dimensional representation feature and a K-dimensional cluster-assignment probability.
3.2. Graph Contrastive (GC)
Graph contrastive learning constructs relationships among samples and encourages representations to follow the graph structure. GCC applies this principle through shared-parameter representation and assignment heads.
- A graph is represented by vertices and an adjacency matrix describing its edge set.
- Graph contrastive learning encourages connected samples to have close representations and disconnected samples to have distant representations.
- Minimizing the graph contrastive loss increases intra-community similarity and decreases inter-community similarity.
- GCC uses two heads with shared CNN parameters: RGC learns clustering-friendly features, while AGC produces compact cluster assignments.
3.3. Framework of GCC
GCC applies graph contrastive learning to both representation and assignment learning through two shared-parameter heads. It constructs a graph from moving-averaged features and extends consistency from individual augmentations to neighboring samples and their cluster assignments.
- Framework overview: GCC uses a representation graph contrastive module and an assignment graph contrastive module with shared CNN parameters.The two modules jointly learn discriminative features and cluster assignments in an end-to-end framework.
- Graph construction: Moving averages of epoch representations reduce training fluctuation before constructing the KNN graph.The moving-average coefficient trades off current and past representations.
- Graph construction: The graph is constructed from neighboring representations and converted into a graph Laplacian for contrastive learning.The KNN graph is followed by graph-Laplacian construction.
- Representation graph contrastive: Representation graph contrastive learning encourages linked samples to have similar features and disconnected samples to remain far apart.The method uses augmented-image features and a similarity function based on graph connectivity.
- Assignment graph contrastive: Assignment graph contrastive learning makes cluster assignment distributions consistent for images, augmentations, and graph neighbors.Neighbor augmentations are sampled through the current graph, and their assignment probabilities are used in the assignment loss.
- Objective: A clustering regularization loss is added to reduce the risk that most samples collapse into a minority of clusters.The overall GCC objective combines representation contrastive learning, assignment contrastive learning, and clustering regularization.
3.4. Model Training
GCC is trained end-to-end by repeatedly sampling images and graph neighbors, generating augmentations, computing three losses, and updating both network parameters and the graph.
- Inputs and initialization: The training algorithm takes unlabeled images, the number of epochs, and the number of clusters as input and outputs a deep clustering model.It initializes the graph and model parameters before iterating over epochs.
- Data sampling: Each epoch samples a mini-batch of images and graph neighbors, then generates augmentations for them.Neighbor sampling follows the current graph.
- Loss computation: The procedure computes representation graph contrastive, assignment graph contrastive, and clustering regularization losses.These losses are combined through the overall GCC objective.
- Optimization: GCC updates network parameters with stochastic gradient descent and then updates the graph before continuing training.The graph is refreshed after minimizing the overall loss.
4. Experiments
Experiments evaluate GCC on six image-clustering benchmarks using ACC, NMI, and ARI, comparing it with traditional and deep-learning methods. GCC substantially outperforms competing approaches, while ablations show that both graph contrastive losses contribute to performance.
- Datasets and evaluation: Experiments use six challenging image-clustering datasets and evaluate performance with ACC, NMI, and ARI.The datasets include CIFAR-10/100, STL-10, ImageNet-10, ImageNet-Dogs, and Tiny-ImageNet.
- Compared methods: GCC is compared with traditional clustering methods and a broad set of deep-learning-based clustering approaches.The comparison includes K-means, spectral clustering, agglomerative clustering, NMF, autoencoder variants, DEC, JULE, and DAC.
- Results: GCC significantly surpasses other methods across most benchmarks under ACC, NMI, and ARI.The paper reports that contrastive learning methods produce more discriminative representations and that GCC achieves the strongest results by a large margin.
- Results: 12.9%, 10.5%, and 4.1% are GCC's ACC improvements over DRC on CIFAR-10, CIFAR-100, and STL-10, respectively.These comparisons are against the second-best method DRC.
4.3. Ablation Study
The ablation studies show that graph contrastive losses and cluster regularization each improve clustering, while GCC consistently outperforms comparison methods and learned features are more clustering-friendly than simCLR.
- Effect of Graph Contrastive Loss: Both RGC and AGC improve clustering on CIFAR-10, CIFAR-100, and ImageNet-10, with GCC achieving the best results.These results indicate that both graph contrastive terms contribute to performance.
- Effect of Cluster Regularization Loss: Cluster regularization significantly improves clustering performance by addressing solutions where most samples are assigned to one cluster.Its effect is small on ImageNet-10, where different classes are already well separated.
- Effect of Self-labeling Fine-tuning: GCC outperforms SCAN before and after self-labeling on all three datasets reported by SCAN.The comparison supports GCC’s learned representations and assignments as more clustering-friendly than SCAN’s.
- Superiority of Graph Contrastive: GCC features produce better clustering performance than simCLR features when evaluated with K-means and Spectral Clustering.The analysis uses GCC with only the representation graph contrastive loss for a fair comparison.
- Superiority of Graph Contrastive: GCC’s top-5 nearest-neighbor accuracy improves during training and remains higher than simCLR across K values from 1 to 50.This supports the use of graph contrastive learning for clustering-oriented representations.
- Superiority of Graph Contrastive: Replacing RGC with a multiple-positive contrastive loss yields substantially worse clustering than GCC.The comparison further supports the proposed graph contrastive framework.
4.4. Qualitative Study
Qualitative analyses show that GCC produces more compact within-class representations and better separates different classes, while case studies reveal shape-based grouping errors.
- Visualization of Representations: On CIFAR-10, GCC’s t-SNE representation makes same-class samples more compact and different classes more clearly separated than basic contrastive learning.For example, class-2 samples split into two groups with simCLR but gather together with GCC.
- Case Study: GCC successfully groups same-class ImageNet-10 images despite differences in background and viewing angle.The case study examines four classes across successful, false-negative, and false-positive cases.
- Case Study: The failure cases indicate that GCC mainly learns object shape, grouping different classes with similar patterns and separating same-class samples with different patterns.Without ground-truth labels, examining these failure details remains difficult for unsupervised learning.
5. Conclusion
The paper introduces GCC, a graph contrastive clustering method that lifts instance-level feature consistency to cluster-level consistency. Experiments on six deep clustering benchmarks show that GCC learns clustering-friendly representations and outperforms state-of-the-art methods.
- GCC applies a novel graph contrastive learning framework to clustering.
- GCC lifts instance-level feature consistency to cluster-level consistency by grouping samples and their augmentations within one cluster.
- GCC learns more clustering-friendly representations than basic contrastive learning and outperforms a wide range of state-of-the-art methods.
- Extensive experiments evaluate GCC on six widely adopted deep clustering benchmarks.