PaperlayerView exact revision on arXiv

Source-linked AI summary

A Comprehensive Survey on Deep Clustering: Taxonomy, Challenges, and Future Directions

Sheng Zhou, Hongjia Xu, Zhuonan Zheng, Jiawei Chen, Zhao li, Jiajun Bu, Jia Wu, Xin Wang, Wenwu Zhu, Martin Ester

arXiv:2206.07579v1cs.LGcs.AI

TL;DR

Shallow clustering struggles with unstructured and high-dimensional data, motivating deep clustering’s joint optimization of representation learning and clustering. This survey synthesizes the field through an interaction-based taxonomy, reviews core designs and resources, and identifies limitations and future directions. It reports that mutual-information methods can unify representation learning and clustering, while contrastive representations can benefit clustering.

  • Problem

    Shallow clustering relies on vectorized representations and cannot be directly applied to unstructured or high-dimensional data; deep clustering also faces challenges in learning discriminative representations and unifying both processes.

  • Method

    The survey organizes deep clustering around representation and clustering modules, classifies methods by their interactions, and reviews designs, datasets, metrics, applications, and implementations.

  • Results

    Mutual-information maximization can overcome the gap between representation learning and clustering in a unified framework, while contrastive representations can benefit clustering.

  • Takeaways & Limitations

    The survey provides a taxonomy and consolidated resources for understanding deep clustering and highlights interactions between representation learning and clustering as a basis for future research.

  • Takeaways & Limitations

    Clustering-friendly representation learning can limit generalization to other clustering methods, and the reverse improvement of representation learning through clustering remains understudied.

Abstract

from arXiv · show

Clustering is a fundamental machine learning task which has been widely studied in the literature. Classic clustering methods follow the assumption that data are represented as features in a vectorized form through various representation learning techniques. As the data become increasingly complicated and complex, the shallow (traditional) clustering methods can no longer handle the high-dimensional data type. With the huge success of deep learning, especially the deep unsupervised learning, many representation learning techniques with deep architectures have been proposed in the past decade. Recently, the concept of Deep Clustering, i.e., jointly optimizing the representation learning and clustering, has been proposed and hence attracted growing attention in the community. Motivated by the tremendous success of deep learning in clustering, one of the most fundamental machine learning tasks, and the large number of recent advances in this direction, in this paper we conduct a comprehensive survey on deep clustering by proposing a new taxonomy of different state-of-the-art approaches. We summarize the essential components of deep clustering and categorize existing methods by the ways they design interactions between deep representation learning and clustering. Moreover, this survey also provides the popular benchmark datasets, evaluation metrics and open-source implementations to clearly illustrate various experimental settings. Last but not least, we discuss the practical applications of deep clustering and suggest challenging topics deserving further investigations as future directions.

1 INTRODUCTION

Deep clustering addresses the difficulty of applying shallow methods to unstructured and high-dimensional data by jointly optimizing representation learning and clustering. This survey organizes the field through a taxonomy based on module interactions and provides resources, applications, and future directions.

  • Motivation: Shallow clustering assumes well-shaped latent vector representations, limiting its direct use for unstructured and high-dimensional data.Deep representation learning has expanded clustering to such data types.
  • Motivation: Deep clustering jointly optimizes deep representation learning and clustering to address representation, unified optimization, and mutual interaction challenges.The survey identifies three central challenges: learning discriminative representations, efficiently unifying both processes, and enabling iterative enhancement.
  • Contributions: The survey proposes a taxonomy of state-of-the-art deep clustering methods based on interactions between representation learning and clustering modules.It also compares the properties of four representative branches across scenarios.
  • Contributions: It summarizes the representation and clustering modules as deep clustering cornerstones and organizes their representative designs for generalization to new models.The survey emphasizes universal designs for both modules.
  • Resources and organization: The survey compiles benchmark datasets, evaluation metrics, open-source implementations, applications, and future research directions.Its resources include references organized on GitHub, while future directions address limitations and challenges.
  • Scope and organization: The survey focuses on interactions between deep representation learning and clustering, while initialization and automatic cluster-number identification are discussed as future work.Comparisons with surveys on shallow clustering, deep clustering, and representation learning appear in Table 1.

2 PRELIMINARY

Deep clustering applies deep neural networks to cluster unstructured or high-dimensional data, with outputs distinguished as hard labels or soft assignment probabilities. The preliminary section also distinguishes partitioning from overlapping clustering.

  • Definitions: Deep clustering uses deep neural networks to cluster unstructured or high-dimensional data, unlike shallow clustering, which operates directly on feature vectors.The definition emphasizes interaction between clustering and deep representation learning.
  • Hard and soft clustering: Hard clustering outputs a discrete one-hot label for each instance, whereas soft clustering outputs a continuous K-dimensional assignment-probability vector.Most deep clustering methods use soft outputs because discrete assignments are difficult to optimize with backpropagation.
  • Hard and soft clustering: For evaluation, a hard label can be obtained from soft clustering by selecting the dimension with maximum probability.This converts the neural network's softmax output into a discrete cluster assignment.
  • Partitioning and overlapping clustering: Partitioning clustering assigns each instance to one of K disjoint groups, while overlapping clustering permits an instance to belong to multiple clusters.Most existing deep clustering methods address disjoint datasets because those datasets dominate evaluation.

3 REPRESENTATION LEARNING MODULE

Deep clustering representation learning modules transform complex data into representations suited to clustering. The survey covers general unsupervised methods, clustering-oriented objectives, and their strengths and limitations.

  • Auto-Encoder based Representation Learning: Auto-encoders encode inputs into low-dimensional representations and decode them back to reconstructed data.They can be customized with architectures for vectorized features, images, graphs, and videos.
  • Auto-Encoder based Representation Learning: Auto-encoder representations are easy to implement and train but largely ignore relationships among different instances.This can leave embedded instances insufficiently discriminated for clustering.
  • Deep Generative Representation Learning: Deep generative representation learning models data as generated from latent representations and infer posterior representations from observations.Variational auto-encoders optimize an evidence lower bound using a prior and variational posterior.
  • Mutual Information Maximization Representation Learning: Mutual information measures dependence between variables and can compare instances with cluster assignments despite differing dimensions or semantic spaces.Neural estimators such as Jensen-Shannon divergence support its use in unsupervised representation learning.
  • Mutual Information Maximization Representation Learning: Mutual-information objectives remain instance-wise, but their sample-dependent marginal distributions capture relationships among instances implicitly.The survey associates this implicit relational information with improved deep-clustering performance.
  • Contrastive Representation Learning: Contrastive learning pulls positive pairs together and pushes negative pairs apart, producing representations with alignment and uniformity properties.Augmentations can connect instances from the same category, while negative samples come from other instances or a memory bank.
  • Clustering Friendly Representation Learning: Clustering-friendly representations directly optimize embeddings to scatter around cluster centroids under the K-means assumption.Their direct clustering optimization can improve corresponding performance, but their specificity limits generalization to other clustering methods.

4 CLUSTERING MODULE

Deep clustering modules transform learned representations into cluster assignments while designing interactions that jointly improve representation learning and clustering. Major strategies include relation matching, pseudo labeling, self-training, mutual-information maximization, and contrastive learning.

  • Representative clustering module: Deep clustering modules map low-dimensional representations to hard labels or soft cluster-assignment probabilities, often reducing features to k dimensions before softmax.These modules are designed to interact with deep representation learning rather than operate as detached shallow clustering algorithms.
  • Relation matching: Relation matching aligns continuous relationships between instances in embedding and label spaces to bridge representation learning and clustering.It is straightforward to implement, but comparing N^2 instance pairs is computationally inefficient; nearest-neighbor or high-confidence relations reduce this cost.
  • Pseudo labeling: Pseudo labeling uses high-confidence instance labels or must-link/cannot-link relations to provide supervised guidance for representation learning.Its performance depends strongly on pseudo-label quality, which is sensitive to model capability and hyper-parameter tuning, motivating pre-training in some methods.
  • Self-training: Self-training minimizes KL divergence between the cluster-assignment distribution Q and an auxiliary distribution P normalized across instances and clusters.Squared assignment probabilities emphasize confident instances, while soft cluster frequencies help prevent assigning all instances to one cluster.
  • Mutual information and contrastive learning: Mutual-information maximization and contrastive learning connect representation learning with clustering in unified objectives, with augmentation improving clustering robustness.Contrastive methods pull positive pairs together and push negative pairs apart, using cluster assignments or other pair definitions.

5 TAXONOMY OF DEEP CLUSTERING

The survey organizes deep clustering methods into four branches according to how representation learning and clustering modules interact: sequentially, iteratively, generatively, or simultaneously.

  • Taxonomy overview: The taxonomy classifies deep clustering by the interaction between representation learning and clustering modules.This interaction-based organization defines the survey’s four methodological branches.
  • Multi-stage deep clustering: Multi-stage deep clustering sequentially connects representation learning with clustering after separately optimizing the two modules.
  • Iterative deep clustering: Iterative deep clustering alternately updates representation learning and clustering modules.
  • Generative deep clustering: Generative deep clustering models the clustering module as a prior representation module.
  • Simultaneous deep clustering: Simultaneous deep clustering updates representation learning and clustering modules at the same time.

5.1 Multistage Deep Clustering

Multistage deep clustering separately optimizes representation learning and clustering in sequence, offering simple deployment and interpretation but limiting information exchange between the modules.

  • Definition: Multistage methods first learn representations with deep unsupervised techniques and then apply classic clustering to those representations.This sequential detachment facilitates clustering analysis and implementation.
  • Representative approaches: Early approaches combined deep autoencoders with K-means, while deep subspace clustering learned affinity matrices or representations before spectral clustering or K-means.
  • Representative approaches: Recent multistage methods incorporate clustering priors into representation learning before clustering the resulting target-friendly representations.IDFD learns instance similarities and reduces within-feature correlations, after which naive K-means can achieve competitive results across existing methods.
  • Advantages and limitations: Multistage methods provide fast deployment, programming-friendly implementation, and intuitive understanding.
  • Advantages and limitations: Sequential cascades cut off information flow between representation learning and clustering, allowing limitations in either component to affect final performance.Cluster structure can guide representation learning, but the cascade does not provide that reciprocal interaction.

5.2 Iterative Deep Clustering

Iterative deep clustering alternates between producing clustering results and updating representations, enabling mutual promotion through pseudo-label supervision but risking error propagation.

  • Definition: Iterative methods repeatedly calculate clustering results from current representations and update representations using the current clustering results.Their motivation is that representations benefit clustering while clustering results provide supervision for representation learning.
  • Supervision: Individual supervision uses pseudo labels generated by the clustering module to train representation learning in a supervised manner.
  • Representative approaches: Early methods updated centroids and assignments in a K-means manner, while later methods combined neural representation learning and clustering in one network.Representative systems also derive pseudo labels through spectral clustering, pseudo-similarity graphs, or neural self-labeling.
  • Supervision: Relational supervision constructs pairwise guidance from pseudo labels, using same-label relations or similarities between assignment probabilities.These relations train the representation module through binary classification or related objectives.
  • Advantages and limitations: Iterative methods benefit from mutual promotion but suffer error propagation when inaccurate early clustering produces chaotic representations and weak self-labeling.

5.3 Generative Deep Clustering

Generative deep clustering combines latent cluster modeling with neural networks to cluster complex data while modeling or generating observations. The survey covers mixture-based, VAE-based, and GAN-based approaches, along with their benefits and limitations.

  • Generative models: Generative models infer clustering assignments by estimating data density under hypotheses about latent cluster structure.Gaussian Mixture Models assume data are generated from a mixture of Gaussians and can learn parameters and assignments with expectation maximization.
  • Deep generative models: Deep generative models combine generative models with deep neural networks to capture nonlinear patterns in complex data.The survey discusses applications to images, texts, graphs, and other complex data types.
  • VAE-based clustering: VaDE and GMVAE stack mixture models with neural networks that generate latent vectors and transform them into complex data instances.Their parameters and cluster assignments are learned by maximizing data likelihood, with VAE variational inference used for tractable optimization.
  • Variants: Variants improve generative clustering through augmentation consistency, mixture-model optimization, architectural changes, and clustering-specific losses.Examples include constraining original and augmented instances to share clustering distributions and introducing an inverse network with a clustering-specific loss.
  • GAN-based clustering: GAN-based models use a generator, discriminator, and mixture-of-Gaussians prior to fit complex data distributions containing multiple clusters.The survey states that equipping GANs with multimodal probability distributions can provide a better fit when data include many different clusters.
  • Limitations: Generative deep clustering can incur unstable, computationally expensive training and inherit limitations from VAE and GAN models.VAE-based approaches may also require prior assumptions about data distributions that may not hold in real data.

5.4 Simultaneous Deep Clustering

Simultaneous deep clustering jointly optimizes representation learning and clustering end-to-end, organizing methods around self-training, mutual information, contrastive learning, and hybrids. Its unified optimization produces clustering-oriented representations but can create optimization imbalance and degenerate solutions.

  • Overview: Simultaneous deep clustering optimizes representation learning and clustering simultaneously in an end-to-end framework.Unlike iterative methods, its two modules are not optimized in an explicitly alternating manner.
  • Self-training: DEC combines an autoencoder with self-training, using a pretrained encoder to initialize joint representation learning and clustering optimization.IDEC extends this approach by adding reconstruction loss to preserve local instance structure.
  • Self-training: Self-training variants improve robustness and address noisy, augmented, subspace, unbalanced, and out-of-distribution data settings.StatDEC modifies the target distribution with normalized cluster instance frequency to preserve discrimination of small groups and form local boundaries less sensitive to imbalance.
  • Mutual information: Mutual-information methods encourage consistency between deep and shallow representations and extend that consistency to cluster assignments.DCCM encourages instances with the same pseudo-labels to share similar representations.
  • Contrastive learning: Contrastive clustering pulls similar instances together and separates different clusters, while preserving augmentation invariance and local robustness.CC treats each cluster as an instance in a low-dimensional space and transfers instance discrimination to cluster discrimination.
  • Hybrid methods: Hybrid methods combine contrastive representation learning, self-training, relation matching, pseudo-labeling, autoencoders, spectral clustering, and augmentation.The survey also describes methods replacing negative-sample contrastive learning with BYOL to reduce reliance on negative samples.
  • Limitations: Unified optimization can bias attention between representation learning and clustering and may lead models toward degenerate solutions.The survey identifies manually setting a balancing parameter as the current mitigation for optimization-focus imbalance.

6 DATASETS AND EVALUATION METRICS

The survey reviews benchmark datasets across image, text, video, and graph clustering, and describes standard metrics for evaluating clustering validity. ACC and NMI range from 0 to 1, whereas ARI ranges from −1 to 1, with higher values indicating better performance.

  • Datasets: Video clustering commonly uses Kinetics-400, Kinetics-600, UCF-101, and HMDB-51 for tasks ranging from action classification to video anomaly detection.The cited datasets are presented as widely used video benchmarks.
  • Datasets: Graph clustering uses network datasets for node clustering and PROTEINS and MUTAG for graph-level clustering evaluation.The survey points to commonly used graph datasets and the Stanford Network Analysis Project as references for node-clustering data.
  • Evaluation metrics: Accuracy, Normalized Mutual Information, and Adjusted Rand Index are the three standard deep-clustering performance metrics reviewed.ACC measures average correct classification rate, NMI measures mutual information between predicted and ground-truth labels, and ARI measures pairwise clustering decisions adjusted for chance.
  • Evaluation metrics: ACC and NMI ∈[0, 1], while ARI ∈[−1, 1], and higher values indicate better clustering performance.ARI adjusts the Rand Index because random partitions do not yield a constant value approaching zero.

7 APPLICATIONS

Deep clustering supports downstream applications including community detection, anomaly detection, image segmentation, medical imaging, and gene-data clustering. The survey also identifies financial, trajectory, and social-media analysis as potential application domains.

  • Community Detection: Deep clustering supports community detection by representing graph nodes in a low-dimensional space for node-level clustering.Community detection partitions graph networks into subgraphs mainly according to connection density.
  • Anomaly Detection: Deep clustering can form a better clustering space for anomaly detection and can be integrated with anomaly identification and removal.Recent approaches jointly identify and remove anomalous instances to reduce their impact on clustering, while improved clustering can further improve anomaly detection.
  • Image Segmentation: Deep clustering has been applied to image segmentation by using clustered regions to generate scene graphs.Image segmentation divides pixels into disjoint regions and can be viewed as unsupervised pixel clustering.
  • Medical Imaging: Deep clustering automatically categorizes large-scale medical images, addressing the labor and expertise demands of manual labeling.The survey mentions medical image clustering analysis for faster COVID-19 diagnosis.
  • Gene Data: Applications also include single-cell RNA sequencing and gene-data clustering, with methods used to discover novel cell types.Examples include DEC for scRNA-seq clustering and MARS, which combines transfer learning and clustering.
  • Other Domains: Financial analysis, trajectory analysis, and social-media understanding are identified as additional domains with potential for deep clustering.The survey links this potential to increasing data volume and dimensionality.

8 FUTURE DIRECTIONS

The survey identifies future directions spanning initialization, clustering scope, representation learning, explainability, transfer learning, outlier handling, and the trade-off between efficient training and global modeling.

  • Initialization: Pre-training-based initialization may be inappropriate for deep clustering because representation learning and clustering are jointly modeled by deep networks.The survey highlights initialization as important for training efficiency and stability in this setting.
  • Clustering Scope: Existing methods largely assume partitioning clustering, while real-world instances may belong to multiple clusters, communities, or tags.The survey calls for greater attention to overlapping or multi-membership clustering scenarios.
  • Representation Learning: Future work should study how clustering can in turn improve representation learning, beyond using representation learning primarily to boost clustering.The survey emphasizes that clustering structure captures high-order dataset patterns that comprehensive representations should preserve.
  • Explainability: Clustering explainability remains difficult because unsupervised clustering usually lacks human priors such as label semantics and the number of clusters.User-provided tags can help, but practical use depends on accurate human tagging.
  • Transfer Learning with Deep Clustering: Transfer learning with deep clustering must address distribution shift and determine how clustering results can transfer knowledge while reducing that shift.The survey describes bidirectional benefits between deep clustering and unsupervised transfer learning, while identifying effective knowledge transfer as open.
  • Outlier Handling: Most deep clustering methods lack a specific response to outliers, even though anomalies can influence and mutually restrict clustering.The survey connects this issue to the known sensitivity of classic K-means to outliers.
  • Efficient Training VS Global Modeling: Mini-batch training improves efficiency and scalability but may lose global modeling ability because deep clustering relies on relationships among instances.Cluster representations or prototypes can store global information, but balancing efficiency and model capability remains open.

9 CONCLUSION

The survey presents an up-to-date overview of deep clustering, organizing methods by how representation learning and clustering interact. It also compiles evaluation resources, applications, and future research opportunities.

  • Conclusion: The survey summarizes deep clustering’s representation learning and clustering modules and proposes a taxonomy based on their interaction.The taxonomy includes multi-stage, iterative, generative, and simultaneous deep clustering.
  • Conclusion: It collects benchmark datasets, evaluation metrics, applications, and future directions to characterize the research field.
Loading 2206.07579v1…