PaperlayerView exact revision on arXiv

Source-linked AI summary

Pseudo-supervised Deep Subspace Clustering

Juncheng Lv, Zhao Kang, Xiao Lu, Zenglin Xu

arXiv:2104.03531v1cs.CVcs.AIcs.LGcs.MM

TL;DR

DSC faces a tension between reconstruction that can overlook relations among points and an n × n self-expression layer that limits scale. PSSC combines relation-weighted reconstruction with pseudo-label supervision, and the authors report average gains over DSC across three clustering metrics.

  • Problem

    DSC self-reconstruction can overlook relations between neighboring points, and its n × n self-expression layer creates a memory cost that hinders large-scale applications.

    Source: §I · pp. 1–2

  • Method

    PSSC uses learned pairwise relations to weight reconstruction and a classifier to generate pseudo-labels that supervise feature and graph learning.

    Source: §I · p. 2

  • Results

    PSSC enhances DSC by 4.48% in ACC, 4.74% in NMI, and 3.73% in purity on average.

    Source: D. Results · p. 7

  • Takeaways & Limitations

    The authors demonstrate a way to address large-scale and out-of-sample subspace clustering using latent representations and a nearest-neighbor classifier.

    Source: B. Setup · p. 9, §VI · p. 10

Abstract

from arXiv · show

Auto-Encoder (AE)-based deep subspace clustering (DSC) methods have achieved impressive performance due to the powerful representation extracted using deep neural networks while prioritizing categorical separability. However, self-reconstruction loss of an AE ignores rich useful relation information and might lead to indiscriminative representation, which inevitably degrades the clustering performance. It is also challenging to learn high-level similarity without feeding semantic labels. Another unsolved problem facing DSC is the huge memory cost due to $n\times n$ similarity matrix, which is incurred by the self-expression layer between an encoder and decoder. To tackle these problems, we use pairwise similarity to weigh the reconstruction loss to capture local structure information, while a similarity is learned by the self-expression layer. Pseudo-graphs and pseudo-labels, which allow benefiting from uncertain knowledge acquired during network training, are further employed to supervise similarity learning. Joint learning and iterative training facilitate to obtain an overall optimal solution. Extensive experiments on benchmark datasets demonstrate the superiority of our approach. By combining with the $k$-nearest neighbors algorithm, we further show that our method can address the large-scale and out-of-sample problems.

I. INTRODUCTION

Deep subspace clustering (DSC) depends on latent representations and similarity graphs, but self-reconstruction can distort relationships between points and the self-expression layer incurs substantial memory cost. PSSC combines weighted reconstruction with pseudo-supervision to address these challenges, with the paper reporting benchmark gains and a 100,000-point DSC demonstration.

II. RELATED WORK

  • Prior subspace-clustering work explores affinity regularizers, joint graph learning, latent-space graphs, and DSC self-expression layers, but concrete supervision remains a challenge.Source: §I · pp. 1–2

I. INTRODUCTION

  • Self-reconstruction can overlook relations between neighboring points, while DSC’s n × n self-expression layer creates a memory cost that hinders large-scale use.The introduction also notes that clustering and reconstruction can have conflicting aims: clustering removes nondiscriminative details, while reconstruction can lose discriminative information.Source: §I · pp. 1–2
  • PSSC combines DSC with iterative pseudo-supervision and weights reconstruction by learned pairwise relations to preserve local structure.A classifier uses latent representations and the similarity graph to construct pseudo-labels that supervise feature and graph learning.Source: §I · p. 2
  • The authors claim their approach is the first to handle 100,000 data points for DSC.Source: §I · p. 2
  • The paper reports that extensive experiments verify the superiority of its method compared with numerous state-of-the-art subspace models.Source: §I · p. 2

A. Deep Clustering

The section traces deep subspace clustering’s reliance on self-expression and explains how reconstruction that ignores relationships among samples can weaken representations. PSSC combines locality-preserving reconstruction, self-expression-based similarity learning, and pseudo-supervision in a jointly connected framework.

A. Deep Clustering

  • DSC adds a self-expression layer between encoder and decoder, using sample combinations to model subspace relations, but its reconstruction loss ignores local structure.The self-expression layer’s weights represent coefficient matrix C; the method overview notes that existing DSC variants use self-reconstruction loss and ignore local structure.Source: A. Deep Clustering · p. 3, A. Base Model · p. 4

B. Pseudo-supervised Learning

  • Pseudo-supervision uses pseudo-labels rather than ground-truth labels, but unreliable labels can mismatch real labels and mislead data grouping.Prior clustering approaches also use pseudo-labels or pseudo-graphs to guide representation learning.Source: B. Pseudo-supervised Learning · p. 3

III. PROPOSED METHOD

  • PSSC jointly combines DSC with pseudo-supervised learning so information from latent representations and self-expression can guide feature and similarity learning.Its locality-preserving, self-expression, and pseudo-supervision modules are interconnected and mutually enhance one another.Source: B. Pseudo-supervised Learning · p. 3, §III · p. 3

A. Base Model

  • The locality-preserving module weights AE reconstruction of each sample by its similarity to other samples, assigning lower similarity to more distant pairs.The method learns the similarity matrix from data through self-expression rather than relying on predefined values.Source: A. Base Model · p. 3
  • The self-expression module represents each data point as a linear combination of others, with coefficients modeling sample relations and an optional zero diagonal excluding C = I.Its loss regularizes C and minimizes reconstruction error in latent space; the self-expression layer is a bias-free fully connected layer between encoder and decoder.Source: A. Base Model · p. 4

B. Pseudo-Supervision Module

The pseudo-supervision module uses a learned similarity graph and classifier predictions to guide training. It supplements pairwise graph supervision with confident pseudo-labels, while assigning probabilities to pairs rather than selecting them through a threshold.

Pseudo-Graph Supervision

  • The module uses pseudo-graph and pseudo-label supervision to guide network training, exploiting similarity information learned from the data.The classifier produces pseudo-labels from the latent representation, and its prediction distribution is encouraged to agree with the learned similarity graph.Source: B. Pseudo-Supervision Module · p. 4
  • Unlike DAC and DCCM, PSSC learns its pseudo-graph from latent representations and assigns each sample pair a probability instead of thresholding pairs.The authors note that threshold selection can be difficult and may include noisy false-positive pairs.Source: B. Pseudo-Supervision Module · p. 4

Pseudo-Label Supervision

  • Only samples with highly confident pseudo-labels contribute to pseudo-label supervision, using a threshold on prediction probability.The paper motivates this selection because predictions may not be one-hot due to non-convexity.Source: B. Pseudo-Supervision Module · p. 4

C. The Unified Formulation

PSSC combines its component losses into a unified objective and trains the network by back-propagation. After optimization, it yields a lower-dimensional representation and a relation matrix.

C. The Unified Formulation

  • The final PSSC objective combines losses (4), (5), (7), and (10) into one jointly optimized formulation.The formulation includes the network parameters for the encoder, self-expression layer, classifier, and decoder.Source: C. The Unified Formulation · pp. 4–5
  • Back-propagation optimizes the network, producing the lower-dimensional representation Z and relation matrix C.The decoder output depends on the encoder, self-expression coefficients, and decoder parameters.Source: C. The Unified Formulation · p. 5

IV. SUBSPACE CLUSTERING EXPERIMENTS

The section evaluates the effectiveness of the proposed PSSC method.

IV. SUBSPACE CLUSTERING EXPERIMENTS

  • The experiments evaluate the effectiveness of the proposed PSSC method.Source: §IV · p. 5

A. Datasets

The experiments use five subspace-clustering benchmark datasets spanning face and object images, compare PSSC with shallow and deep clustering methods, and include an ablation of pseudo-supervision.

A. Datasets

  • The evaluation covers five commonly used, relatively high-dimensional subspace-clustering datasets: two face datasets and three object datasets.The dataset descriptions include ORL, MNIST, Umist, and COIL20; the fifth dataset is named inconsistently in the supplied passages.Source: A. Datasets · p. 5

B. Comparison Methods

  • Comparisons include shallow subspace-clustering methods and deep clustering approaches, including DSC-L1, DSC-L2, DEC, DKM, DCCM, DEPICT, and DSCDAN.The listed shallow methods include LRR, LRSC, SSC, KSSC, SSC-OMP, and EDSC.Source: B. Comparison Methods · p. 5
  • The ablation removes Lgraph and Llabel from the objective to form PSSCl, isolating a model without those pseudo-supervision terms.Source: B. Comparison Methods · p. 5

C. Setup

The setup pre-trains an autoencoder, then fine-tunes the full PSSC network and constructs an affinity matrix for spectral clustering. Clustering is evaluated with ACC, NMI, and purity.

C. Setup

  • The encoder and decoder use one convolutional layer on COIL20 and COIL40, and three layers on the other datasets, with ReLU activations.The network settings are summarized in Table II.Source: C. Setup · p. 5, Table II · p. 6
  • Training pre-trains the encoder and decoder, then fine-tunes the full network with grid-searched γ1, γ2, and γ3; pseudo-label confidence uses a threshold of 0.8.The learning rates are 1e-3 for pre-training and 1e-4 for fine-tuning; Adam is used.Source: C. Setup · p. 5
  • The learned relation matrix C is converted into affinity matrix A for spectral clustering, with α selected according to noise and q denoting maximal subspace intrinsic dimension.The affinity construction is summarized in Algorithm 2.Source: C. Setup · pp. 5–6
  • Clustering performance is assessed with accuracy (ACC), normalized mutual information (NMI), and purity (PUR).Source: C. Setup · p. 6

D. Results

PSSC performs strongly across benchmark comparisons, with reported gains over DSC and improvements associated with pseudo-supervision and locality-preserving reconstruction. The results also compare learned affinities and reconstructions with DSC.

D. Results

  • PSSC achieves the best performance among the comparison methods on the benchmark datasets.Table III reports clustering results across the benchmark datasets.Source: D. Results · p. 6
  • PSSC considerably outperforms shallow subspace-clustering methods, which the authors attribute mainly to neural networks’ representation ability.Source: D. Results · p. 6
  • PSSC improves over DSC by 4.48% in ACC, 4.74% in NMI, and 3.73% in purity, on average; PSSCl’s improvement over DSC is presented as verifying the importance of locality preservation.Source: D. Results · pp. 6–7
  • PSSC always outperforms PSSCl, and pseudo-supervision boosts ACC by 5.8% on MNIST and 6.46% on Umist.The authors say pseudo-supervision guides feature and relation learning.Source: D. Results · p. 7
  • PSSC is reported as comparable to DeepSC and S2ConvSCN-ℓ2, though those methods have high time complexity for the stated reasons.The authors report DeepSC’s ℓ1 problem per epoch and spectral clustering in S2ConvSCN-ℓ2; DEPICT and DSCDAN are described as inferior to PSSC.Source: D. Results · p. 7
  • On MNIST, PSSC reconstructions preserve edges better than DSC and reach PSNR 15.54 versus 15.04; its affinity matrix also has more block-diagonal energy and fewer non-diagonal points.The affinity comparison is visualized in Fig. 2; reconstructed images are shown in Fig. 3.Source: Figure 2 · p. 6, D. Results · p. 7, Figure 3 · p. 7

E. Parameter Analysis

Parameter analyses examine sensitivity to γ2 and γ3, the assumed cluster count k, and intrinsic dimension q. The paper also introduces large-scale experiments to address the n × n self-expression-layer memory cost.

E. Parameter Analysis

V. LARGE-SCALE EXPERIMENTS

  • Large-scale experiments are introduced to mitigate the n × n self-expression-layer size, using MNIST, USPS, and RCV1.The experiments use complete MNIST and USPS datasets and sample 10,000 documents from four large RCV1 categories, represented by 2,000 high-tf-idf words.Source: §V · p. 7, A. Datasets · pp. 7, 9

A. Datasets

  • Table V is identified as reporting clustering results on MNIST, USPS, and RCV1, but the supplied passage gives no result values.Source: Table V · p. 9

B. Setup

PSSC is trained on a small sample, then uses nearest-neighbor classification to assign labels to the remaining data. The experiments use a shared encoder/decoder architecture for comparison with selected deep-clustering methods.

B. Setup

  • PSSC trains on 5,000 samples per dataset, clusters their affinity matrix, then labels the remaining encoded samples with a nearest-neighbor classifier.This setup is used to address large-scale and out-of-sample problems.Source: B. Setup · pp. 9–10, Figure 10 · p. 10
  • For fair comparison, PSSC uses the same fully connected encoder/decoder architecture as DEC, IDEC, and DKM.The encoder dimensions are d-500-500-2000-k, and the decoder mirrors them as k-2000-500-500-d.Source: B. Setup · p. 9
  • The large-scale workflow samples a training batch, pre-trains the auto-encoder, fine-tunes PSSC, and applies spectral clustering before assigning labels to the rest.The algorithm denotes the sampled training data Xc and the remaining data Xu.Source: B. Setup · pp. 9–10
  • The comparison includes classical and deep-clustering baselines, while DCCM, DEPICT, kSCN, and DSCDAN are omitted on text data because they target images.The study uses the SVD version of kSCN, reported as its best-performing version in.Source: B. Setup · p. 10

C. Results

PSSC reports higher ACC than several deep-clustering baselines across the evaluated datasets, with exceptions against DEPICT on MNIST. USPS visualizations show better class separation for PSSC, and the conclusion identifies large-scale and out-of-sample clustering as demonstrated capabilities.

C. Results

  • PSSC improves ACC over DKM by 5%, 17.8%, and 19.7% across three datasets, and over kSCN by 2% on MNIST and 6% on USPS.The results section also reports more than 23% ACC improvement over DCCM.Source: C. Results · p. 10
  • DEPICT's MNIST ACC is approximately 3.9% higher than PSSC's, while PSSC's USPS ACC is 7.9% higher.The paper attributes PSSC's performance to subspace embeddings and pseudo-supervised learning.Source: C. Results · p. 10
  • On USPS t-SNE visualizations, PSSC separates classes, while IDEC and DKM merge classes and DEPICT, DSCDSN, and kSCN show cluster separation problems.The reported issues include a displaced black class for DEPICT and close or blended clusters for DSCDSN and kSCN.Source: C. Results · p. 10

VI. CONCLUSION

  • PSSC unifies locality preservation, self-expression, and self-supervision; future work will explore other self-supervision approaches and complex sample distributions.The conclusion reports experiments on small- and large-scale datasets and demonstrates large-scale and out-of-sample clustering.Source: §VI · p. 10
Loading 2104.03531v1…