Source-linked AI summary
Deep Spectral Clustering using Dual Autoencoder Network
Xu Yang, Cheng Deng, Feng Zheng, Junchi Yan, Wei Liu
TL;DR
The paper tackles the limited latent-dependency modeling of shallow clustering and the weak connection between reconstruction loss and discriminative embeddings. It combines a dual autoencoder, mutual-information estimation, and deep spectral clustering in a jointly optimized framework, and reports superior performance across benchmark datasets, including over 98% accuracy on MNIST-test.
Problem
Shallow methods rely on low-level features and local relationships, while reconstruction loss alone does not ensure discriminative latent representations.
Method
A dual autoencoder reconstructs clean and noisy latent representations, mutual-information estimation preserves discriminative input information, and deep spectral clustering jointly assigns latent features to clusters.
Results
The method outperforms state-of-the-art approaches across benchmark datasets, achieving over 98% ACC on MNIST-test and exceeding DEPICT by 1.6% ACC and 3.1% NMI.
Takeaways & Limitations
Jointly learning robust, discriminative representations with spectral clustering improves clustering performance across the evaluated benchmark datasets.
Abstract
from arXiv · showhide
The clustering methods have recently absorbed even-increasing attention in learning and vision. Deep clustering combines embedding and clustering together to obtain optimal embedding subspace for clustering, which can be more effective compared with conventional clustering methods. In this paper, we propose a joint learning framework for discriminative embedding and spectral clustering. We first devise a dual autoencoder network, which enforces the reconstruction constraint for the latent representations and their noisy versions, to embed the inputs into a latent space for clustering. As such the learned latent representations can be more robust to noise. Then the mutual information estimation is utilized to provide more discriminative information from the inputs. Furthermore, a deep spectral clustering method is applied to embed the latent representations into the eigenspace and subsequently clusters them, which can fully exploit the relationship between inputs to achieve optimal clustering results. Experimental results on benchmark datasets show that our method can significantly outperform state-of-the-art clustering approaches.
1. Introduction
The paper addresses limitations of shallow clustering and reconstruction-based deep clustering by jointly learning robust, discriminative embeddings and spectral clusters. Its dual autoencoder, mutual-information estimation, and deep spectral clustering framework outperforms state-of-the-art methods across five benchmark datasets.
- Shallow clustering relies on low-level features and local distances that inadequately represent latent dependencies among inputs.
- Reconstruction loss alone does not guarantee discriminative latent representations, creating a performance gap for deep clustering.
- The dual autoencoder reconstructs latent representations and their noisy versions to generate more robust embeddings.
- Mutual-information estimation preserves discriminative information from inputs, while deep spectral clustering exploits relationships among latent representations.
- The dual autoencoder and deep spectral clustering network are jointly optimized in a unified learning framework.
- Experiments on five benchmark datasets show that the method outperforms state-of-the-art traditional and deep clustering models.
2. Related Work
Related work includes conventional, deep generative, convolutional, adversarial, and continuous clustering methods. The paper distinguishes its approach by combining relative reconstruction, mutual-information estimation, and joint optimization with deep spectral clustering.
- Deep Embedding Clustering learns latent representations with a stacked autoencoder and minimizes reconstruction loss during pre-training.
- Deep generative approaches include variational autoencoders with Gaussian-mixture priors and methods that modify autoencoder objectives for clustering.
- Continuous clustering methods jointly optimize dimensionality reduction, cluster representatives, or convex clustering objectives.
- JULE, DEPICT, VaDE, self-expressive autoencoders, and DASC combine neural representation learning with agglomerative, softmax, mixture, sparse, or adversarial clustering mechanisms.
- The proposed method uses relative reconstruction and mutual-information estimation, then jointly optimizes the autoencoder and deep spectral clustering networks.
3. Methodology
The framework jointly learns robust, discriminative latent representations and spectral-clustering assignments. A dual autoencoder combines clean and noise-disturbed reconstruction with mutual-information estimation, while spectral clustering exploits sample relationships in an eigenspace.
- Discriminative latent representation: The dual autoencoder reconstructs inputs from both latent representations and noise-disturbed versions to improve robustness.The complete reconstruction objective combines original and relative reconstruction losses, with δ controlling their strengths.
- Discriminative latent representation: Mutual-information estimation preserves information shared by inputs and latent representations, using discriminator-based negative sampling for global and local estimates.Local estimation relates a convolutional middle-layer feature map to the latent representation through a 1×1 convolution.
- Discriminative latent representation: The latent distribution is regularized toward a standard normal prior, while JS divergence replaces unbounded KL divergence for mutual-information maximization.The paper states that the prior constraint makes the latent space more regular.
- Deep Spectral Clustering: The learned latent representations initialize a deep spectral-clustering network that maps them into the eigenspace of an associated graph Laplacian.The method constructs a non-negative affinity matrix from the latent representations before applying the spectral-clustering loss.
- Deep Spectral Clustering: An orthogonalization layer enforces the spectral-clustering constraint by transforming minibatch outputs through QR- and Cholesky-based operations.The layer produces orthogonalized outputs using a linear map and the required √m scaling factor.
- Deep Spectral Clustering: Latent-representation learning and spectral clustering are unified through KL divergence and jointly optimized until convergence.The network output is constrained to be orthonormal in expectation to prevent all points from collapsing into one cluster.
4. Experiments
Experiments evaluate the method on five benchmark datasets using ACC and NMI, compare it with conventional and deep clustering baselines, and examine parameter sensitivity and training strategies.
- Datasets and metrics: The study evaluates clustering on five benchmark image datasets using Accuracy (ACC) and Normalized Mutual Information (NMI).The evaluation includes MNIST-full, MNIST-test, USPS, and Fashion-MNIST among the selected datasets.
- Comparisons: The method is compared with conventional algorithms, including K-means, spectral clustering, NMF, and agglomerative clustering, alongside deep learning-based approaches.The compared deep methods include DASC and DEC.
- Clustering performance: The proposed method outperforms competing methods across the benchmark datasets and exceeds DEPICT on MNIST-test by 1.6% ACC and 3.1% NMI.Its clustering accuracy on MNIST-test is over 98%.
- Parameter sensitivity: Parameter-sensitivity results show acceptable performance for most β and γ combinations, indicating relative stability on MNIST-test.Figure 5 reports ACC and NMI for different parameter settings.
- Training strategies: Ablation results show that mutual information and improved reconstruction loss each enhance clustering accuracy, especially when combined.The compared strategies include ConvAE, ConvAE+MI, ConvAE+RS, and ConvAE+MI+RS.
- Representation visualization: t-SNE visualizations show clearer distribution structure in latent representations produced by the proposed model than in the compared representations.The visualizations include raw data, convolutional-autoencoder representations, DEC, SpectralNet, and proposed strategies.
5. Conclusion
The paper proposes an unsupervised deep clustering framework combining a dual autoencoder with a deep spectral network. It improves latent representations through noise robustness and information preservation, then exploits sample relationships for clustering.
- The dual autoencoder reconstructs inputs from both latent representations and noise-contaminated versions to obtain more robust features.
- Mutual-information maximization preserves input information in the latent representations, making the learned features more discriminative.
- A fused spectral network clusters latent-space features while effectively utilizing relationships between samples.
- Experiments on several benchmark datasets show that the method outperforms state-of-the-art approaches.