PaperlayerView exact revision on arXiv

Source-linked AI summary

Model-based clustering based on sparse finite Gaussian mixtures

Gertraud Malsiner-Walli, Sylvia Frühwirth-Schnatter, Bettina Grün

arXiv:1606.06828v1stat.ME

TL;DR

The paper tackles joint estimation of mixture-component number and cluster-relevant variables in Bayesian finite Gaussian mixtures. It uses sparse hierarchical priors with MCMC, then identifies the model through relabeling. The approach improves component-mean estimation and identifies relevant variables, but its cluster interpretation is limited for non-Gaussian data.

  • Problem

    Finite-mixture clustering must estimate an unknown number of clusters and identify variables whose heterogeneity carries the cluster structure.

  • Method

    The approach deliberately overfits a finite Gaussian mixture, uses sparse priors on weights and component means, and applies MCMC followed by relabeling for identified inference.

  • Results

    The normal gamma shrinkage prior improves component-mean estimates and identifies cluster-relevant variables, while sparse weights empty superfluous components during MCMC.

  • Takeaways & Limitations

    A finite Gaussian-mixture framework can jointly estimate components, classify observations, identify relevant variables, and support component-specific inference using standard MCMC methods.

  • Takeaways & Limitations

    The strategy was investigated under multivariate Gaussian components; for skewed or fat-tailed clusters, multiple Gaussian components may represent one cluster, so non-empty-component counts lose their cluster interpretation.

Abstract

from arXiv · show

In the framework of Bayesian model-based clustering based on a finite mixture of Gaussian distributions, we present a joint approach to estimate the number of mixture components and identify cluster-relevant variables simultaneously as well as to obtain an identified model. Our approach consists in specifying sparse hierarchical priors on the mixture weights and component means. In a deliberately overfitting mixture model the sparse prior on the weights empties superfluous components during MCMC. A straightforward estimator for the true number of components is given by the most frequent number of non-empty components visited during MCMC sampling. Specifying a shrinkage prior, namely the normal gamma prior, on the component means leads to improved parameter estimates as well as identification of cluster-relevant variables. After estimating the mixture model using MCMC methods based on data augmentation and Gibbs sampling, an identified model is obtained by relabeling the MCMC output in the point process representation of the draws. This is performed using $K$-centroids cluster analysis based on the Mahalanobis distance. We evaluate our proposed strategy in a simulation setup with artificial data and by applying it to benchmark data sets.

1 Introduction

The paper addresses joint estimation of the unknown number of mixture components and cluster-relevant variables within Bayesian finite Gaussian mixtures. It proposes sparse priors and MCMC-based inference while retaining a finite-mixture framework.

  • 1 Introduction: Finite-mixture clustering must estimate the unknown number K of data clusters, and no uniquely best selection method has been identified.Existing approaches include BIC, related model-choice criteria, RJMCMC, marginal likelihoods, and infinite-mixture methods.
  • 1 Introduction: Unnecessary variables can mask cluster structure, motivating simultaneous identification of cluster-relevant variables.The paper notes that heterogeneity may occur only in a subset of available variables.
  • 1 Introduction: The paper proposes sparse finite mixtures as a Bayesian alternative to infinite mixtures without assuming the number of components or relevant variables are known.The framework deliberately overfits the finite mixture and induces sparsity through priors on weights and component means.
  • 1 Introduction: A Dirichlet prior on mixture weights is chosen so that superfluous components are emptied during MCMC sampling.The prior hyperparameters are selected according to asymptotic results requiring values smaller than d/2, where d is the component-parameter dimension.
  • 1 Introduction: A normal gamma shrinkage prior on component means identifies cluster-relevant variables while pulling means together in homogeneous dimensions.The resulting estimates are described as more precise, and relevant variables can be distinguished by dispersion of cluster locations.
  • 1 Introduction: The framework also addresses component-specific inference by resolving label switching in overfitting mixtures through MCMC permutation and post-processing relabeling.Identified component parameters are needed for inference about cluster-specific quantities.

2 Model specification

The model uses sparse priors on mixture weights and component means to estimate non-empty components and identify cluster-relevant variables. The normal gamma prior provides flexible shrinkage, while the Dirichlet prior controls redundant components.

  • 2.1 Identifying the number of mixture components: The model assigns a symmetric Dirichlet prior to mixture weights in an intentionally overfitting finite mixture.The number of fitted components K exceeds the true number, so the prior must encourage redundant components to empty.
  • 2.1 Identifying the number of mixture components: When the Dirichlet hyperparameter e0 is smaller than d/2, superfluous components asymptotically receive weights converging to zero.For e0 greater than d/2, overfitting can instead produce identical components with non-negligible weights.
  • 2.1 Identifying the number of mixture components: Finite-sample applications may require much smaller e0 values than the asymptotic condition e0 < d/2 suggests.The paper considers either a very small fixed value or a gamma hyperprior G(a, b) for learning the required sparsity.
  • 2.1 Identifying the number of mixture components: The number of components is estimated by the posterior mode of K0, the number of non-empty components visited during MCMC sampling.The posterior distribution of K0 is estimated from relative frequencies across MCMC iterations.
  • 2.2 Identifying cluster-relevant variables: The standard prior independently models component means as μk ∼ N(b0, B0), with hyperparameters based on the data median and variable ranges.A shrinkage prior is preferred when some dimensions are expected to have homogeneous component means.
  • 2. Model specification: The Gaussian mixture model leaves variance-covariance matrices unconstrained in geometric shape rather than imposing sparse covariance structure.The paper uses a conjugate hierarchical prior for these matrices.

3 Bayesian estimation

The Bayesian estimator uses sparse finite Gaussian mixtures with MCMC to estimate non-empty components and address component identification. Normal-gamma shrinkage improves mean behavior but can increase empty-component allocations, motivating stronger weight shrinkage.

  • Bayesian estimation: MCMC estimation augments the model with latent allocations and alternates classification with conditional simulation of component parameters and weights.Observations are allocated using ηk fN(yi|μk, Δk), after which μk, Δk, and ηk are simulated conditional on allocations.
  • Bayesian estimation: Random permutation steps make the MCMC sampler explore all K! labelings and avoid trapping around a single posterior mode.This addresses the component-label symmetry of finite mixtures during posterior sampling.
  • Shrinkage and empty components: Normal-gamma shrinkage places empty-component means near the data center or non-empty component locations, increasing their subsequent allocation probabilities.Under the standard prior, empty-component means can disperse far from the data, making empty components likely to remain empty.
  • Shrinkage and empty components: In the two-component simulation, the superfluous component is more dispersed under the standard prior but lies near true means or the data center under the normal-gamma prior.The corresponding allocation probability for the superfluous component is considerably higher under the normal-gamma prior.
  • Shrinkage and empty components: Because normal-gamma shrinkage can overestimate non-empty components, the Dirichlet prior uses a very small fixed e0 to shrink small weights more strongly.Smaller e0 produces smaller weights for empty components.

4 Identifying sparse finite mixtures

Identification removes label ambiguity by clustering posterior component-parameter draws after filtering sparse-mixture output. Mahalanobis K-centroids clustering better captures elliptical posterior structure than Euclidean K-means.

  • Label switching: Finite-mixture identification requires post-processing because reordering components leaves the mixture representation unchanged.Symmetric priors make component-specific posterior inference difficult without a unique labeling.
  • Point-process representation: The point-process representation displays posterior component-parameter draws independently of label switching and supports model identification.Two-dimensional projections produce scatter plots of component-specific MCMC draws.
  • Relabeling: Clustering the retained draws yields a unique labeling for draws whose classifications are permutations, enabling component-specific parameter inference.The classification obtained from posterior means can also reorder covariance matrices and weights.
  • Point-process representation: For sparse mixtures, empty-component draws are removed and only iterations with the estimated number of non-empty components are clustered.In the Crabs data, retaining iterations with four non-empty components clearly separates the posterior means of the four non-empty components.
  • Mahalanobis K-centroids: Mahalanobis distance captures elongated elliptical posterior clusters, whereas squared-Euclidean K-means can split one cluster and combine fragments into an artificial cluster.The improved fit reduces the non-permutation rate to 0 and improves component-specific parameter inference.
  • Mahalanobis K-centroids: Mahalanobis K-centroids clustering uses cluster-specific centroids and positive-definite dispersion matrices to minimize assigned distances.Because no closed-form solution exists, an iterative assignment-and-update algorithm is run until convergence to a local optimum.

5 Simulations and applications

Simulations and applications show that sparse priors can recover mixture components, improve mean estimation, identify cluster-relevant variables, and support identified-model inference.

  • Simulations: Under the standard prior, the estimated number of non-empty components matched the true number in all equal-weight data sets for K = 15 and K = 30.Exactly four components were non-empty for most MCMC draws, and the non-permutation rate was zero.
  • Simulations: For equal-weight data, combining the normal gamma prior with a very small fixed e0 emptied superfluous components and recovered four groups, even with K = 30.The required value was e0 = 0.001 for K = 30; e0 = 10^-5 also estimated four groups.
  • Simulations: The normal gamma prior reduced MSEμ to ≈0.136 versus ≈0.167 under the standard prior, while misclassification rates were about equal across K.This indicates more efficient estimation of component-specific means in the equal-weight simulation.
  • Component selection: The estimator remained robust to a very small component because it counted components that stayed non-empty during MCMC rather than selecting only large weights.The fourth component was never emptied, although its weight was very small and could be confused with a superfluous component.
  • Variable selection: The normal gamma prior identified the first two variables as cluster-generating in both simulation settings, including when the first component was small.Posterior shrinkage factors λ_j were strongly pulled toward zero for non-cluster-generating variables in the equal-weight setup.
  • Applications: For the Crabs data, Mahalanobis-distance K-centroids clustering captured elliptical posterior geometry and reduced the non-permutation rate to 0, unlike squared Euclidean distance.The refined relabeling procedure improved inference for component-specific parameters, as reflected in comparisons of MSEμ.

6 Discussion

The paper proposes sparse finite Gaussian mixtures to jointly estimate component number, component parameters, classifications, and cluster-relevant variables using standard MCMC. It reports practical benefits alongside limitations concerning non-Gaussian data, visual variable identification, and high-dimensional sampling.

  • Sparse priors on mixture parameters enable simultaneous estimation of the unknown component number, component-specific parameters, observation classifications, and cluster-relevant variables.
  • A sparse prior on mixture weights empties superfluous components during MCMC, allowing the number of true components to be estimated from the number of non-empty components without marginal-likelihood calculations or sophisticated RJMCMC proposals.
  • The strategy was studied under multivariate Gaussian component assumptions, with non-Gaussian extensions suggested for nonsymmetrical cluster shapes and outliers.
  • For skewed or fat-tailed mixtures, multiple Gaussian components may represent one cluster, so the estimated non-empty-component count may no longer represent distinct clusters.
  • A normal gamma prior on component means supports cluster-relevant-variable identification and more precise component-location estimates than the standard prior.
  • Visual identification of cluster-relevant variables may be cumbersome with many variables, while high-dimensional Gibbs sampling can become trapped in local modes and fail to empty superfluous components.

Appendix 1: Scheme for estimating using MCMC sampling

The MCMC scheme alternates parameter simulation, observation classification, and hyperparameter sampling, with additional normal-gamma updates and random label permutations.

  • The sampling scheme begins with parameter simulation conditional on the current classification.
  • For each observation, the algorithm samples its component classification conditional on the current means, precisions, and mixture weights.
  • The procedure samples hyperparameters, including the Dirichlet prior hyperparameter e0 through a Metropolis-Hastings step when it is random.
  • Under the normal gamma prior, the sampler includes an additional normal-gamma-specific update.
  • A random permutation of the K component labels is applied to component means and precisions to address label switching.

Appendix 2: Scheme for clustering in the point process representation

Post-processing estimates the component count from MCMC occupancy, clusters component-mean draws using Mahalanobis distance, and retains identified draws for inference.

  • For each MCMC iteration, the procedure determines the number of non-empty components.
  • The estimated component count is the mode of the number of non-empty components observed during MCMC sampling.
  • Only iterations with the estimated non-empty-component count are retained, and draws from empty components are removed.
  • All remaining component-mean draws are clustered into the estimated number of groups using K-centroids analysis with Mahalanobis distance.
  • The resulting classifications define iteration-specific permutation sequences used to relabel component-specific draws.
  • Non-permutation draws are removed, while a high non-permutation fraction indicates overlapping point-process clusters and typically signals an overfitted mixture model.
Loading 1606.06828v1…