PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Clustering: A Comprehensive Survey

Yazhou Ren, Jingyu Pu, Zhimeng Yang, Jie Xu, Guofeng Li, Xiaorong Pu, Philip S. Yu, Lifang He

arXiv:2210.04142v1cs.LG

TL;DR

Deep clustering addresses the challenge that shallow clustering depends on representative features, while existing surveys give insufficient attention to data sources and initial conditions. This survey systematically classifies deep clustering methods by those factors, analyzes their advantages and applicable conditions, and identifies future research directions.

  • Problem

    Shallow clustering models are effective only when features are representative, motivating deep clustering methods that learn data representations.

  • Method

    The survey organizes deep clustering methods by data sources and initial settings, alongside deep-learning methodology and network or data structure.

  • Results

    The survey analyzes the advantages, disadvantages, and applicable conditions of different deep clustering methods and presents research directions.

  • Takeaways & Limitations

    Deep clustering should be studied with attention to the characteristics and conditions of the clustering data.

  • Takeaways & Limitations

    Pixel-GAN relies heavily on generated-image quality and is difficult to apply to complex images.

Abstract

from arXiv · show

Cluster analysis plays an indispensable role in machine learning and data mining. Learning a good data representation is crucial for clustering algorithms. Recently, deep clustering, which can learn clustering-friendly representations using deep neural networks, has been broadly applied in a wide range of clustering tasks. Existing surveys for deep clustering mainly focus on the single-view fields and the network architectures, ignoring the complex application scenarios of clustering. To address this issue, in this paper we provide a comprehensive survey for deep clustering in views of data sources. With different data sources and initial conditions, we systematically distinguish the clustering methods in terms of methodology, prior knowledge, and architecture. Concretely, deep clustering methods are introduced according to four categories, i.e., traditional single-view deep clustering, semi-supervised deep clustering, deep multi-view clustering, and deep transfer clustering. Finally, we discuss the open challenges and potential future opportunities in different fields of deep clustering.

1 INTRODUCTION

Deep clustering addresses the limited feature-learning power of shallow clustering on complex data. This survey organizes methods by data sources and initial conditions, covering single-view, semi-supervised, multi-view, and transfer settings.

  • Shallow clustering methods work effectively with representative features but are usually limited on complex data because their feature-learning power is poor.
  • Deep clustering uses deep networks to extract clustering-friendly features from complex data, leveraging nonlinear mapping capability and flexibility across scenarios.
  • Existing deep-clustering surveys mainly emphasize single-view tasks and neural-network architectures, leaving broader application scenarios less covered.
  • The survey classifies deep clustering according to data sources and initial conditions, analyzing methods, prior knowledge, architectures, advantages, disadvantages, and applicable conditions.
  • Semi-supervised deep clustering incorporates limited prior constraints through constraint losses, while multi-view clustering exploits consistent and complementary information across representations.
  • Transfer deep clustering uses information from relevant tasks to improve clustering when the target task has limited instances or high dimensionality.

2 DEFINITIONS AND PRELIMINARIES

The survey formalizes deep clustering as learning a map from data samples to k clusters. It distinguishes single-view, semi-supervised, multi-view, and domain-adaptation settings by their data and constraints.

  • Given data samples X, deep clustering seeks a map function F that maps X into k clusters represented by ˆY.
  • Semi-supervised deep clustering: Semi-supervised deep clustering incorporates prior constraints through a constrained matrix A.
  • Deep multi-view clustering: Deep multi-view clustering operates on multiple representations, with X_i denoting the i-th view of X.
  • Deep clustering with domain adaptation: Deep clustering with domain adaptation distinguishes a labeled source domain (X_s, Y_s) from an unlabeled target domain X_t.

3 DEEP SINGLE-VIEW CLUSTERING

Deep single-view clustering is organized by how feature extractors are trained and how clustering objectives are integrated. The survey covers DAE-, DNN-, VAE-, GAN-, and GNN-based approaches, including separate or joint analysis and structure-aware methods.

  • Single-view methods are categorized by feature-extractor training as DAE-, DNN-, VAE-, GAN-, and GNN-based approaches.Their main difference lies in the loss components used to train representations.
  • DAE-based deep clustering: DAE-based methods either cluster low-dimensional encoder features separately or jointly optimize feature extraction and clustering results.Separate analysis uses traditional clustering after embedding, whereas joint analysis combines representation learning with clustering objectives.
  • DAE-based deep clustering: Separate-analysis methods add clustering-relevant constraints, including locality preservation, group sparsity, structure priors, and self-expressive affinity learning.DEN uses locality-preserving and group-sparsity constraints; PARTY and DSC-Nets incorporate global or sample-relationship structure.
  • DAE-based deep clustering: DEC jointly learns embeddings and cluster assignments by first minimizing reconstruction loss, then iteratively optimizing KL(S∥R) with adjusted target assignments.S represents soft assignments and R is a purer target distribution; DEC variants add reconstruction, local-structure, or regularization objectives.
  • DNN-based deep clustering: DNN-based methods require extra tasks to train the feature extractor, while mutual-information and contrastive objectives can improve cluster separation and avoid degenerate solutions.DRC introduces contrastive terms to decrease intra-class variance and increase inter-class variance.
  • GNN-based deep clustering: Structure-aware GNN methods use graph information explicitly; SDCN stacks GNN layers to capture high-order structure while combining autoencoder and GNN self-supervision.The survey identifies SDCN as the first work to apply structural information explicitly in deep clustering.

4 SEMI-SUPERVISED DEEP CLUSTERING

Semi-supervised deep clustering incorporates must-link and cannot-link constraints into neural clustering methods, addressing a setting that remains relatively underexplored. These constraints can be added as loss terms to improve clustering-oriented feature learning.

  • Semi-supervised clustering uses must-link constraints for same-cluster samples and cannot-link constraints for different-cluster samples.
  • Semi-supervised deep clustering combines relationship constraints with neural networks, with representative methods summarized in Table 4.
  • SDEC extends DEC by incorporating pairwise constraints into feature learning through an additional loss term.
  • The constraint coefficient aij is 1 for must-link, -1 for cannot-link, and 0 otherwise, encouraging linked embeddings together and unlinked embeddings apart.The loss makes ||zi−zj||2 small when aij = 1 and large when aij = -1.
  • Unsupervised deep clustering methods can generally be expanded into semi-supervised ones by adding a semi-supervised loss.The added information can help neural networks extract features more suitable for clustering.

5 DEEP MULTI-VIEW CLUSTERING

Deep multi-view clustering addresses data represented in multiple forms by learning from view-specific, shared, complementary, or graph-based structure. The survey organizes these methods into DEC-based, subspace-clustering-based, and GNN-based categories.

  • Multi-view clustering handles inputs with multiple forms, such as different languages, camera angles, or writing styles for the same underlying content.
  • Deep multi-view clustering methods are grouped into DEC-based, subspace-clustering-based, and GNN-based categories according to their clustering basis.Most methods use autoencoders, and the categories are summarized in Table 5.
  • 5.1 DEC-based: DAMC combines a multi-view encoder, generator, discriminators, and DEC layer to learn low-dimensional embeddings and reconstruct each view.Its loss includes reconstruction, clustering, and GAN components.
  • 5.1 DEC-based: DAMC is reported to reveal nonlinear multi-view structure and achieve better clustering performance than traditional MVC algorithms.
  • 5.1 DEC-based: DEMVC uses a switched shared auxiliary target distribution so each view can guide the others toward clustering-conducive low-dimensional features.Reconstruction optimization is also used to retain view discrepancies.
  • 5.1 DEC-based: Contrastive methods such as CMHHC and MFLVC learn cross-view invariance, hierarchical similarity, multiple feature levels, or reduced influence from view-private information.CMHHC supports clustering at diverse granularities, while MFLVC addresses incomplete multi-view clustering through complementary information.
  • 5.2 Subspace clustering-based: Subspace-based multi-view methods address limitations of shallow features and weak cross-view interaction by learning affinity or self-representation structures with deep networks.Subspace clustering estimates pairwise affinities and applies spectral clustering or normalized cut; self-expression represents points using combinations of X.
  • 5.2 Subspace clustering-based: DMVSSC uses CCA-based self-expression and convolutional autoencoders to integrate common latent features and explore complementary information across views.

6 DEEP CLUSTERING WITH TRANSFER LEARNING

Deep transfer clustering addresses data drawn from different feature spaces or distributions by transferring knowledge between source and target domains. The survey distinguishes DNN-based methods, which measure domain discrepancy, from GAN-based methods, which use discriminators to fit that measurement.

  • Motivation: Transfer learning addresses clustering when training and testing data come from different feature spaces or distributions.This setting is especially relevant to complex, high-dimensional data where traditional and deep clustering may perform poorly.
  • DNN-based methods: DNN-based UDA methods project source and target domains into a shared feature space so source-trained classifiers can apply to the target domain.These methods focus on selecting an appropriate measurement of domain discrepancy.
  • DNN-based methods: DAN uses multiple-kernel MMD to minimize distance at feature-specific layers and project source and target features into the same space.The framework freezes early shared-encoder layers while allowing later layers to be fine-tuned.
  • DNN-based methods: Discrepancy-based alternatives align domains using supervised hash and unsupervised entropy losses, sliced Wasserstein discrepancy, second-order statistics, or higher-order moments.These approaches replace MMD with other domain-alignment criteria.
  • GAN-based methods: GAN-based UDA methods use generators to produce data across distributions and discriminators to judge whether generated data follow the target distribution.Co-GAN uses paired GANs with shared weights in semantic layers, encouraging similar high-level representations while retaining different low-level features.
  • Design considerations: Transfer methods must select layers responsible for high-level semantics; DAN applies MMD to later feature-specific layers, while Co-GAN shares semantic layers.GenToAdapt-GAN separates feature extraction from adversarial processing and is described as more stable on complex data than Co-GAN.
  • Design considerations: Pixel-GAN relies heavily on generated-image quality, while CyCADA adapts both pixel- and feature-level representations with structural and semantic consistency.The survey notes that generated-instance methods can perform poorly on real images such as those in the OFFICE dataset.

7 FUTURE DIRECTIONS OF DEEP CLUSTERING

The survey identifies future directions spanning graph-structured data, theory, massive complex datasets, and multi-view fusion. It emphasizes better use of structure and complementary information alongside scalable and theoretically grounded deep clustering.

  • Theoretical exploration: Deep clustering lacks reliable theory explaining how feature extraction and clustering loss influence final clustering outcomes.The survey calls for theoretical analysis of deep clustering optimization.
  • Massive complex data processing: Massive, heterogeneous data create uncertainty because most existing models are designed for specific datasets.The survey points to deep learning and graph learning for complex data processing.
  • Massive complex data processing: Small datasets make deep clustering prone to overfitting, whereas large datasets impose high computational complexity.The survey suggests model-structure optimization and compression to reduce computational load and improve practical efficiency.
  • Fusion of multi-view data: Most current deep clustering algorithms use only one information type, limiting their ability to exploit text, voice, image, and other complementary views together.Future work should integrate two or more views while using their consistency and complementarity.
  • Deep clustering based on graph learning: Graph-based deep clustering should better exploit structural associations and account for differences and importance across views.The survey identifies rational use of complex graph information as an open problem for both single-view and multi-view clustering.

8 SUMMARY OF DEEP CLUSTERING METHODS

The survey organizes deep clustering by data structure and problem setting, covering single-view, semi-supervised, multi-view, and transfer learning methods. It concludes that complex applications motivate broader use of multi-source information.

  • Survey scope: The survey covers single-view, semi-supervised, multi-view, and transfer-learning deep clustering according to different data structures and problem settings.Single-view methods are further introduced according to the neural networks on which they are based.
  • Single-view methods: DAE- and DNN-based methods appeared earlier but are described as limited by poor performance on real datasets.VAE- and GAN-based methods attract attention for feature extraction and sample generation capabilities.
  • Graph-based methods: GNN-based clustering methods are summarized because graph neural networks are prominent in community-discovery problems.The survey also notes their use within multi-view clustering.
  • Problem settings: Semi-supervised clustering adds constraint losses, multi-view clustering supplements information across views, and transfer learning moves knowledge from labeled to unlabeled domains.Transfer methods are categorized into DNN-based measurement strategies and GAN-based discriminator strategies.
  • Practical outlook: Single-view clustering remains challenging for complex data, while semi-supervised, multi-view, and unsupervised domain-adaptation models incorporate multi-source information for practical scenarios.Examples include multiple news organizations, time-frequency sensor signals, and transferring a dog-classification network to cat images.
Loading 2210.04142v1…