PaperlayerView exact revision on arXiv

Source-linked AI summary

Simultaneous model-based clustering and visualization in the Fisher discriminative subspace

Charles Bouveyron, Camille Brunet

arXiv:1101.2374v2stat.MEstat.APstat.COstat.ML

TL;DR

High-dimensional clustering remains challenging for accuracy and understanding, particularly when observations are few relative to variables. The paper proposes Fisher-EM to estimate a parsimonious discriminative mixture model and its subspace; experiments report better clustering than existing methods, with useful representations for real and simulated data.

  • Problem

    High-dimensional clustering is challenging because mixture methods can be over-parameterized, especially when the number of observations is small relative to the number of variables.

    Source: §1 · p. 1, §4.6 · p. 20

  • Method

    Fisher-EM estimates mixture parameters and a latent orthonormal discriminative subspace using a family of 12 parsimonious DLM models.

    Source: §1 · p. 2, §7 · p. 32

  • Results

    Fisher-EM performs better than existing clustering methods on simulated and real datasets, and in mass-spectrometry clustering it provides pertinent interpretation.

    Source: §7 · p. 34

  • Takeaways & Limitations

    When the number of groups is not too large, the estimated discriminative subspace provides a useful projection of clustered data, including in mass-spectrometry analysis.

    Source: §7 · p. 34

    Convergence is proved for only two DLM models; convergence for the other models remains to be investigated.

    Source: §7 · p. 34

Abstract

from arXiv · show

Clustering in high-dimensional spaces is nowadays a recurrent problem in many scientific domains but remains a difficult task from both the clustering accuracy and the result understanding points of view. This paper presents a discriminative latent mixture (DLM) model which fits the data in a latent orthonormal discriminative subspace with an intrinsic dimension lower than the dimension of the original space. By constraining model parameters within and between groups, a family of 12 parsimonious DLM models is exhibited which allows to fit onto various situations. An estimation algorithm, called the Fisher-EM algorithm, is also proposed for estimating both the mixture parameters and the discriminative subspace. Experiments on simulated and real datasets show that the proposed approach performs better than existing clustering methods while providing a useful representation of the clustered data. The method is as well applied to the clustering of mass spectrometry data.

1 Introduction

High-dimensional clustering is difficult because mixture methods can be over-parameterized, especially when observations are few relative to variables. The introduction proposes Fisher-EM to combine clustering with a discriminative subspace, addressing both model estimation and interpretation.

1 Introduction

  • Mixture clustering can perform poorly in high dimensions because its models are over-parameterized, a difficulty amplified when observations are few relative to variables.The introduction identifies this setting in applications including mass spectrometry and genomics.Source: §1 · pp. 1–2
  • Common dimension-reduction methods such as PCA do not account for the clustering task and can discard information that is discriminative.The introduction notes that existing methods combining dimension reduction with classification are supervised.Source: §1 · p. 2
  • Group-specific subspace clustering can be effective but does not provide a global visualization, making clustered results harder to interpret.Interpretation by original-variable meaning can also matter in fields such as genomics and economics.Source: §1 · p. 2
  • Fisher-EM combines a mixture model with a Fisher-based discriminative subspace to target clustering performance, high-dimensional estimation, and low-dimensional representation.The introduction describes the representation as supporting understanding of clustered data.Source: §1 · p. 2
  • The paper reports that convergence is proved only for one DLM model in its introduction, with convergence for the others left for investigation.This is the introduction's stated status of the convergence analysis.Source: §1 · p. 3

2 Related works

The related-work section surveys model-based clustering and approaches to high-dimensional data, including dimension reduction, subspace clustering, and Fisher-based discrimination. It contrasts these approaches in their modeling, classification, and visualization aims.

2 Related works

  • Clustering partitions observations into homogeneous groups, while high-dimensional data motivate approaches that exploit their low-dimensional structure.The section frames the review around model-based clustering and methods for high dimensionality.Source: §2 · p. 3

2.1 Model-based clustering and high-dimensional data

  • When sample size is small relative to dimension, full Gaussian mixtures require many covariance parameters and can encounter ill-conditioned or singular covariance matrices.With fewer observations than dimensions, the full model requires restrictive assumptions.Source: §2.1 · pp. 3–4

2.2 Dimension reduction and clustering

  • PCA projects data onto lower-dimensional axes of high variance, while feature-selection approaches can remove information that may discriminate groups.The cited approaches include PCA and feature-selection work,,, and.Source: §2.2 · p. 4

2.3 Subspace clustering

  • Probabilistic subspace methods model groups in low-dimensional latent spaces and can cluster high-dimensional data efficiently, but give less attention to visualization and understanding.The review distinguishes heuristic searches for dense subspaces from probabilistic generative approaches.Source: §2.3 · pp. 4–5

2.4 From Fisher’s theory to discriminative clustering

  • Fisher discriminant analysis finds a low-dimensional linear transformation that favors large between-class and small within-class covariance, with dimension at most K − 1.Prior clustering approaches combining discriminative spaces with k-means do not compute the discriminant subspace and do not address visualization or understanding.Source: §2.4 · pp. 5–6

3 Model-based clustering in a discriminative subspace

The DLM model fits data in a lower-dimensional discriminative latent subspace, aiming to combine parsimonious clustering with a visualization. Its 12 constrained submodels vary covariance assumptions, while their orientation is chosen to discriminate groups rather than maximize projected variance.

3 Model-based clustering in a discriminative subspace

  • DLM models use a common latent subspace of dimension d ≤ K −1, lower than the observed dimension p, to model and visualize clustered data.The latent space is assumed to be the most discriminative subspace.Source: §3 · p. 6, §3.1 · p. 6

3.1 The discriminative latent mixture model

  • The DLM[Σkβk] model links observed data to latent data through a shared orthonormal subspace, with group-specific latent means and covariances and noise variances.The model parameters include mixture proportions, means, subspace orientation U, within-subspace covariance Σk, and outside-subspace variance βk.Source: §3.1 · pp. 6–7

3.2 The submodels of the DLM[Σkβk] model

  • The 12 DLM submodels arise from constraints on within- and between-group covariance parameters, including shared, diagonal, or isotropic variance assumptions.For example, a common β across groups models the variance outside the discriminative subspace with one parameter.Source: §3.2 · p. 8
  • DLM models have low complexity and modeling capacity comparable to Mixt-PPCA; their complexity depends on K and p, unlike Mixt-PPCA’s dependence on hyper-parameter d.For K = 4 and p = 100, Full-GMM requires 20,603 parameters, versus 803 for Diag-GMM and 407 for Sphe-GMM.Source: Table 1 · p. 9, §3.2 · p. 10

3.3 Links with existing models

  • Unlike common-loading approaches that orient subspaces to maximize projected variance, DLM chooses the orientation to discriminate groups, which should improve clustering and visualization in most cases.The authors expect this distinction to matter when the greatest-variance axes are not parallel to the discriminative axes.Source: §3.3 · pp. 10–11

4 Parameter estimation: the Fisher-EM algorithm

This section presents Fisher-EM for estimating DLM clustering models through alternating posterior assignment, discriminative-subspace estimation, and mixture-parameter updates. It also covers initialization, model selection, convergence, computational cost, visualization and variable interpretation, and adapting the F step to high-dimensional, low-sample settings.

4.1 The E step

  • E-step posterior membership probabilities depend on distances in the discriminative and complementary subspaces, balanced by their variances and mixture proportions.The complementary-space distance can be reformulated using projections onto the discriminative subspace, avoiding projection onto the complement and supporting stability when n ≪ p.Source: §4.1 · p. 12

4.2 The F step

  • The F step constructs orthonormal discriminative axes sequentially, maximizing the Fisher criterion in orthogonal subspaces to address the non-orthogonal axes produced by traditional Fisher methods.The procedure adapts the orthonormal discriminant vector approach to the current soft partition and obtains each axis through a generalized eigenvalue problem.Source: §4.2 · pp. 13–14

4.3 The M step

  • In the M step, mixture proportions and means are updated from empirical counterparts, while variance parameters are estimated for the 12 DLM models.The variance estimates use the discriminative axes from the F step and soft covariance information.Source: §4.3 · pp. 15, 17

4.4 Initialization and model selection

  • A PCA-derived latent space is proposed as a way to adapt mixture-parameter initialization, and the experiments use BIC for model selection while noting ICL may also suit this context.Model selection covers both the number of components and model parametrization.Source: §4.4 · pp. 17–18

4.5 Computational aspects

  • Fisher-EM convergence is guaranteed for DLM[αβ], but not generally for other DLM models; in practice, those models rarely fail to converge when correctly initialized.The section proposes Aitken’s criterion to monitor convergence and an optional Hessian check for whether the estimate is a local maximum.Source: §4.5 · pp. 18–19
  • The iterative F step adds computational cost, but its complexity does not depend on n and is not quadratic in p when d is small; simulations found Fisher-EM 1.5 times slower than diagonal-model EM on average.For 1,000 observations in 100 dimensions, Fisher-EM took an average of 24 seconds versus 16 seconds for EM.Source: §4.5 · p. 19

4.6 Practical aspects

  • The latent dimension is at most K − 1; the authors recommend setting d = K − 1 when numerically possible, and the estimated subspace supports visualization when K ≤ 4.For K > 4, the actual dimension d can be estimated from the eigenvalue scree.Source: §4.6 · pp. 19–20
  • When d ≤ 3, the first d discriminative axes visualize data without discriminative-information loss; when d > 3, use the three most discriminative axes, though poor clustering can yield poor visualization.The paper suggests initializing with the mini-EM strategy or classical EM to avoid poor clustering results.Source: §4.6 · p. 20
  • The loading matrix U can identify the most discriminative original variables by selecting its highest loadings or thresholding loadings to zero below a chosen value.This is especially useful in fields such as biology or economics, where observed variables have practical meaning.Source: §4.6 · p. 20
  • Fisher-EM can handle n ≪ p by modifying its F step to avoid computing the last p − d columns of W and reduce the eigendecomposition from p × p to n × n.The reduced-space procedure determines d ≤ n axes, which are sufficient for Fisher-EM.Source: §4.6 · pp. 20–21

5 Experimental results

The experiments compare Fisher-EM with existing clustering methods on an introductory iris example, simulations, and real datasets. They assess clustering performance, model selection, and whether the estimated discriminative subspace provides a useful visualization.

5.1 An introductory example: the Fisher’s irises

  • On iris, Fisher-EM reaches 98% clustering accuracy with DLM[αkβ], and its partition is almost the same as supervised OLDA’s.The first estimated discriminative axes are similar, with scalar product −0.996; the log-likelihood evolves monotonically to convergence.Source: Figure 3 · p. 22, Table 3 · p. 22, §5.1 · p. 23

5.2 Simulation study: influence of the dimension

  • Fisher-EM remains more effective and stable as observed dimensionality increases, while Full-GMM, PCA-EM, and Com-GMM deteriorate or encounter numerical problems.Mixt-PPCA accuracy falls to 0.5 from dimension 30; Full-GMM does not work beyond dimension 15 and cannot be used at dimension 20 because of covariance singularity.Source: §5.2 · pp. 24–25, Figure 5 · p. 25

5.3 Simulation study: model selection

  • Eight of the 12 DLM models select the correct four components, and BIC is largest for the data-generating DLM[αkβ] model with four components.DLM models fit the data better by BIC than the other compared methods; Fisher-EM’s three estimated discriminative axes separate the four groups, unlike PCA’s first three components.Source: §5.3 · pp. 25, 27, Table 4 · p. 28, Figure 7 · p. 28

5.4 Real data set benchmark

  • Fisher-EM outperforms other methods on most tested UCI datasets, and DLM[.β] models provide satisfying results across all datasets considered.The comparison covers wine, iris, zoo, glass, satimage, and usps358; some missing Table 5 results reflect algorithm non-convergence.Source: §5.4 · p. 29

6 Application to mass spectrometry

The mass-spectrometry application tests Fisher-EM on high-dimensional cancer and control spectra and compares its clustering with PCA-EM and mixture of PPCA. Fisher-EM reports the highest accuracy among the three and provides unsupervised information about discriminative variables.

6 Application to mass spectrometry

  • MALDI mass spectrometry is presented as a non-invasive biochemical technique with applications including biomarker search, tumor progression assessment, and drug-treatment evaluation.The paper identifies early colorectal cancer detection as a promising application and notes that MALDI imaging could in some cases avoid invasive, expensive colonoscopy.Source: §6 · p. 29

6.1 Data and experimental setup

  • MALDI2009 contains 112 spectra: 64 from patients with colorectal cancer and 48 from healthy controls, each originally 16,331-dimensional.The spectra cover mass-to-charge ratios from 960 to 11,163 Da.Source: §6.1 · p. 29
  • The experiment applies Fisher-EM to 6,168 variables covering m/z 960–3,500 Da, following the protocol of.The paper says the remaining dimensions contain no discriminative information and compares Fisher-EM with PCA-EM and mixture of PPCA on this subset.Source: §6.1 · p. 31

6.2 Experimental results

  • Fisher-EM estimates one discriminative axis, compared with four PCA-EM axes and two axes per group for mixture of PPCA.All three methods were asked to divide the dataset into two groups; the task has n ≪ p, and only these three model-based methods can handle it in the paper's comparison.Source: §6.2 · p. 31, §6.1 · p. 31
  • Fisher-EM reaches 91% accuracy, versus 89% for mixture of PPCA, and makes significantly fewer false negatives with an acceptable number of false positives.PCA-EM also produced satisfying results, but made a significant number of false negatives, a concern because classification risk is asymmetric here.Source: §6.2 · pp. 31–32, Table 7 · p. 32
  • The estimated loading matrix identifies discriminative original variables unsupervised; m/z 2800 and 3050 are selected despite a smaller cancer-control difference than at m/z 1350.The loadings express correlations between original variables and the one-dimensional discriminative subspace; peaks indicate variables highly correlated with its axis.Source: §6.2 · p. 32

7 Conclusion and further works

The paper concludes that DLM models data in a lower-dimensional orthonormal discriminative subspace, estimated with Fisher-EM. Experiments show better clustering than existing methods, while convergence is proved for only two DLM models.

7 Conclusion and further works

  • The DLM model fits data in a lower-dimensional latent orthonormal discriminative subspace, with 12 parsimonious variants formed by constraining parameters within and between groups.Fisher-EM estimates both the mixture parameters and the subspace, adapting the Fisher criterion to unsupervised classification under an orthonormality constraint.Source: §7 · pp. 32, 34
  • Experiments on simulated and real datasets show Fisher-EM performs better than existing clustering methods.The paper also reports efficient clustering and pertinent interpretation in its high-dimensional mass-spectrometry application.Source: §7 · p. 34
  • When the number of groups is not too large, the estimated discriminative subspace provides a useful projection of the clustered data.The paper identifies visualization in 2D or 3D for estimated subspaces exceeding four dimensions as a possible extension.Source: §7 · p. 34
  • Convergence is proved for only two DLM models; the other models still require investigation.The authors suggest convergence might be proved for those models in a generalized EM context.Source: §7 · p. 34
  • Further proposed extensions include a kernel Fisher-EM and sparsity in the loading matrix to ease interpretation of discriminative axes.The kernel proposal would replace the Gram matrix with a kernel; sparsity would use an ℓ1 penalty.Source: §7 · p. 34

A Appendix

The appendix defines a decomposition of W into components associated with the latent discriminative and complementary spaces.

A Appendix

  • The appendix writes W as the sum of W̃ and W̄, where W̃ contains the first d vectors of W, zero-padded, and W̄ is the remainder.W̃ = [U, 0_{p−d}] and W̄ = W − W̃ = [0_d, V].Source: A Appendix · p. 34

A.1 E step

The E step expresses group membership through posterior probabilities and reformulates its cost using the DLM assumptions. The appendix also defines projections onto the discriminative and complementary spaces.

A.1 E step

  • The conditional expectation t_ik is the posterior probability of observation y_i belonging to group k, computed using Bayes’ formula.The passage also relates this posterior to the Gaussian density and the mixture parameters estimated in the previous iteration.Source: §A.1 · pp. 34–35
  • The cost function Γ_k is reformulated from −2 log(π_k φ(y_i, θ_k)) under the DLM[Σ_kβ_k] assumptions and the decomposition W = W̃ + W̄.The derivation uses the constructed matrices’ stated relations and a norm on the latent space.Source: §A.1 · p. 35
  • P projects onto the discriminative space E, while P⊥ projects onto its complement E⊥; the latter gives the complementary-space residual distance.The passage rewrites that distance as the norm of the centered observation minus its projection onto E.Source: §A.1 · p. 35

A.2 M step

The M step maximizes the conditional expectation of the complete log-likelihood to estimate DLM parameters. The appendix gives parameter-specific estimates across several covariance-constraint models.

A.2 M step

  • The M step estimates DLM parameters by maximizing the conditional expectation of the complete log-likelihood.The expectation is evaluated at iteration q using the preceding parameter estimate.Source: §A.2 · p. 36
  • Group prior probabilities are estimated under a sum-to-one constraint, using the soft group-membership expectations t_ik.The text defines t_ik as E[z_ik|θ^(q−1)].Source: §A.2 · pp. 36–37
  • The latent-space group means are estimated by maximizing the expected complete log-likelihood and setting its derivative to zero.Source: §A.2 · p. 38
  • Covariance and constraint parameters receive model-specific estimates, with several DLM variants reusing the β estimates from Equation (A.25).The appendix lists variants including DLM[Σ_kβ_k], DLM[Σβ], and models with α parameters.Source: §A.2 · pp. 38–39
Loading 1101.2374v2…