Source-linked AI summary
Variational Deep Embedding: An Unsupervised and Generative Approach to Clustering
Zhuxi Jiang, Yin Zheng, Huachun Tan, Bangsheng Tang, Hanning Zhou
TL;DR
Clustering needs representations that capture data structure while retaining generative capability. VaDE combines a VAE with a GMM, and it outperforms state-of-the-art clustering methods while generating realistic cluster-conditioned samples without supervision.
Problem
Existing deep clustering methods such as DEC learn representations and assignments but do not model the data-generating process or generate samples.
Method
VaDE combines a GMM and neural networks in a VAE framework, using a mixture-of-Gaussians prior and ELBO optimization with SGVB and reparameterization.
Results
VaDE outperforms state-of-the-art clustering methods by a large margin on 5 benchmarks from different modalities and generates realistic samples conditioned on cluster information without supervised training.
Takeaways & Limitations
VaDE provides an unsupervised generative clustering framework that can both cluster data and generate samples for specified clusters.
Takeaways & Limitations
The paper focuses on clustering, although VaDE can also be used for unsupervised feature learning or semi-supervised learning.
Abstract
from arXiv · showhide
Clustering is among the most fundamental tasks in computer vision and machine learning. In this paper, we propose Variational Deep Embedding (VaDE), a novel unsupervised generative clustering approach within the framework of Variational Auto-Encoder (VAE). Specifically, VaDE models the data generative procedure with a Gaussian Mixture Model (GMM) and a deep neural network (DNN): 1) the GMM picks a cluster; 2) from which a latent embedding is generated; 3) then the DNN decodes the latent embedding into observables. Inference in VaDE is done in a variational way: a different DNN is used to encode observables to latent embeddings, so that the evidence lower bound (ELBO) can be optimized using Stochastic Gradient Variational Bayes (SGVB) estimator and the reparameterization trick. Quantitative comparisons with strong baselines are included in this paper, and experimental results show that VaDE significantly outperforms the state-of-the-art clustering methods on 4 benchmarks from various modalities. Moreover, by VaDE's generative nature, we show its capability of generating highly realistic samples for any specified cluster, without using supervised information during training. Lastly, VaDE is a flexible and extensible framework for unsupervised generative clustering, more general mixture models than GMM can be easily plugged in.
1 Introduction
VaDE addresses clustering by combining deep representation learning with an unsupervised generative model. It uses a GMM and neural networks to learn cluster-oriented embeddings, generate observations, and outperform established methods across multiple datasets.
- Method: The generative process selects a cluster, samples a latent representation from it, and decodes that representation into an observation.A separate encoder maps observations to latent embeddings for ELBO optimization.
- Method: VaDE combines a VAE with a GMM to form an unsupervised generative clustering framework.Its mixture-of-Gaussians prior replaces VAE’s single Gaussian prior, making the latent structure suited to clustering.
- Method: VaDE optimizes the ELBO with the SGVB estimator and reparameterization trick.These enable variational inference through the encoder network.
- Results: VaDE generates highly realistic samples for any specified cluster without supervised information during training.This generative capability distinguishes it from clustering approaches that cannot model the data-generating process.
2 Related Work
Related work spans similarity-based, feature-based, deep clustering, and deep generative approaches. VaDE differs by integrating clustering directly into a VAE through a mixture prior and jointly modeling cluster assignments with latent representations.
- Deep clustering: DEC jointly learns representations and cluster assignments but cannot model the data-generating process or generate samples.Its embeddings are designed specifically for clustering and do not readily extend beyond that task.
- Deep generative models: Deep generative models such as GANs and VAEs capture data distributions from which unseen samples can be generated.Their variants have also been applied to semi-supervised classification and clustering.
- VaDE’s distinction: VaDE replaces VAE’s single Gaussian prior with a Gaussian mixture prior and jointly considers cluster assignments and latent representations.This differs from DLGMM, which uses a Gaussian mixture as the approximate posterior and does not model the class variable.
- VaDE’s distinction: VaDE’s GMM models clustering, whereas GM-CVAE uses its GMM to model transitions between video frames.The two frameworks therefore combine VAE and GMM for different purposes.
3 Variational Deep Embedding
VaDE formulates unsupervised probabilistic clustering within a VAE by combining a Gaussian mixture prior with neural-network encoding and decoding. It optimizes an ELBO and obtains cluster assignments from the learned latent representation.
- 3.1 The Generative Process: VaDE generates observations by selecting a cluster with a categorical GMM, sampling a latent representation, and decoding it with a neural network.The model supports binary or real-valued observations through corresponding Bernoulli or Gaussian likelihoods.
- 3.1 The Generative Process: The joint generative distribution factorizes as p(x, z, c) = p(x|z)p(z|c)p(c), with x and c conditionally independent given z.
- 3.2 Variational Lower Bound: Training maximizes the evidence lower bound with the SGVB estimator and reparameterization trick, using Monte Carlo samples from q(z|x).For binary observations, the ELBO formulation uses the corresponding observation model; the real-valued case is analogous.
- 3.2 Variational Lower Bound: VaDE uses a neural network g to model q(z|x), enabling variational inference over latent embeddings from observed data.The variational posterior is assumed to be mean-field factorized.
- 3.3 Understanding the ELBO of VaDE: The ELBO combines a reconstruction term with KL regularization that encourages latent embeddings to lie on a mixture-of-Gaussians manifold.The cluster posterior q(c|x) is computed through p(c|z), which captures the relationship between cluster assignments and latent representations.
- 3.3 Understanding the ELBO of VaDE: VaDE outperforms AE+GMM and VAE+GMM, supporting the importance of the KL regularization and joint optimization of the VAE and GMM.Figure 2 compares clustering accuracy during MNIST training with several baselines, while Figure 3 shows increasingly cluster-suitable latent representations.
4 Experiments
VaDE is evaluated on five multimodal benchmarks using clustering accuracy and latent-representation comparisons, alongside qualitative analyses of generation and embeddings.
- Experimental Setup: VaDE is evaluated on MNIST, HHAR, Reuters-10K, Reuters, and STL-10 using comparisons with multiple clustering baselines.The setup uses the same network architecture as DEC for a fair comparison.
- Quantitative Comparison: 94.46%, 84.46%, 79.83%, 79.38%, and 84.45% ACC are reported on MNIST, HHAR, Reuters-10K, Reuters, and STL-10, respectively.These results outperform DEC by relative increases of 12.05%, 5.76%, 7.41%, 4.96%, and 4.75%, respectively.
- Quantitative Comparison: VaDE’s unsupervised training is evaluated without comparison to supervised models, despite its learned representations being discriminative.The authors perform 10 random restarts and select the result with the best objective value.
- Generating Samples by VaDE: VaDE generates smooth and diverse MNIST digits for specified classes, with performance comparable to InfoGAN.VAE-generated sample classes cannot be specified in the comparison.
- Visualization of Learned Embeddings: On MNIST embeddings, VaDE has fewer incorrectly clustered samples than VAE and DEC, with VaDE errors mostly near cluster borders.DEC errors often occur inside clusters, which the authors associate with failure to preserve inherent data structure.
- The Impact of the Number of Clusters: When K differs from the number of classes, VaDE groups visually similar digits together for smaller K and separates digit subclasses for larger K.Examples include 9 with 4 and 3 with 8 when K=7, and distinct 0 or 1 styles when K=14.
5 Conclusion
VaDE embeds probabilistic clustering in a VAE, combines a GMM with a neural network, and outperforms state-of-the-art methods while generating cluster-conditioned samples. The framework can also adopt other mixture models.
- VaDE embeds probabilistic clustering into a Variational Auto-Encoder framework using a GMM and neural network.
- VaDE outperforms state-of-the-art methods by a large margin across five benchmarks from different modalities.
- VaDE generates highly realistic samples conditioned on cluster information without supervised training.
- Other mixture models can be adopted flexibly in the VaDE framework.
Appendix A
Appendix A derives the variational cluster posterior q(c|x) by relating it to p(c|z) and optimizing the ELBO. The derivation yields a posterior matching condition.
- The appendix derives q(c|x) as the expectation of p(c|z) under q(z|x).
- Maximizing the ELBO with respect to q(c|x) requires D_KL(q(c|x)||p(c|z)) = 0.
- The derivation takes expectations over the latent representation to obtain the cluster posterior expression.
Appendix B
Appendix B states and proves a lemma for the KL divergence between two diagonal-covariance multivariate Gaussian distributions. The result is expressed elementwise across the latent dimensions.
- Lemma 1 considers q(z) and p(z) as multivariate Gaussians with isotropic diagonal covariance.
- The lemma provides the KL-divergence result used for Gaussian latent distributions.
- The parameters are indexed by latent dimension j, whose total count is J.
Appendix C
Appendix C rewrites VaDE's ELBO into computable expectation terms and explains their SGVB approximation. Reparameterization makes the latent-sampling estimate differentiable with respect to encoder parameters.
- The reconstruction expectation supports either multivariate Bernoulli or multivariate Gaussian observations.
- SGVB approximates the reconstruction expectation using L Monte Carlo samples, with L allowed to equal 1.
- The reparameterization trick produces a differentiable estimate when latent variables are sampled from the encoder distribution.
- The Gaussian KL calculations use latent dimensionality J and cluster count K.
- q(c|x) is computed from the appendix derivation and approximated using SGVB and the reparameterization trick.