PaperlayerView exact revision on arXiv

Source-linked AI summary

Mixture models with a prior on the number of components

Jeffrey W. Miller, Matthew T. Harrison

arXiv:1502.06241v1stat.ME

TL;DR

Mixture models need a principled way to handle an unknown number of components, but MFMs have been harder to use than DPMs because DPM samplers are more readily available. The paper shows that MFMs share core DPM representations and inference properties, enabling direct use of many DPM methods. Simulations and applications demonstrate consistent MFM component-number behavior, distinct clustering behavior, and practical use with high-dimensional gene-expression data.

  • Problem

    Choosing the number of components is difficult, and MFM inference has commonly relied on reversible jump methods despite the broader availability of DPM samplers.

  • Method

    The paper derives exchangeable partition, urn/restaurant-process, random discrete-measure, and sometimes stick-breaking representations for MFMs.

  • Results

    MFMs and DPMs are consistent for density estimation, but MFMs are consistent for the number of components while DPM cluster posteriors fail to concentrate and tend to include tiny extra clusters.

  • Takeaways & Limitations

    Many DPM inference techniques can be directly applied to MFMs without reversible jump MCMC, simplifying MFM use in new applications.

  • Takeaways & Limitations

    Component- and cluster-number inference can be severely affected by model misspecification, with posteriors potentially diverging as sample size increases.

Abstract

from arXiv · show

A natural Bayesian approach for mixture models with an unknown number of components is to take the usual finite mixture model with Dirichlet weights, and put a prior on the number of components---that is, to use a mixture of finite mixtures (MFM). While inference in MFMs can be done with methods such as reversible jump Markov chain Monte Carlo, it is much more common to use Dirichlet process mixture (DPM) models because of the relative ease and generality with which DPM samplers can be applied. In this paper, we show that, in fact, many of the attractive mathematical properties of DPMs are also exhibited by MFMs---a simple exchangeable partition distribution, restaurant process, random measure representation, and in certain cases, a stick-breaking representation. Consequently, the powerful methods developed for inference in DPMs can be directly applied to MFMs as well. We illustrate with simulated and real data, including high-dimensional gene expression data.

1. Introduction

MFMs place a prior on the number of components, but historically lacked the convenient inference machinery available for DPMs. This paper shows that MFMs share key DPM mathematical properties, allowing many DPM inference methods to be applied directly while retaining important differences in clustering and component-number behavior.

  • Motivation: MFMs address the difficulty of choosing a finite mixture’s number of components by placing a prior on that number.Mixture models have applications including population structure, document modeling, speaker recognition, computer vision, phylogenetics, and gene expression profiling.
  • Motivation: DPMs became popular partly because generic MCMC algorithms are easy to adapt and exploit elegant partition, restaurant-process, random-measure, and stick-breaking properties.These properties support broadly applicable inference procedures.
  • Contribution: MFMs typically exhibit analogous exchangeable partition, urn/restaurant-process, random discrete-measure, and sometimes stick-breaking representations.The key properties hold for any prior distribution on the number of components.
  • Contribution: Most inference methods developed for DPMs can therefore be used for MFMs without reversible jump MCMC, simplifying applications across conjugate and non-conjugate settings.Examples include Jain–Neal split-merge samplers coupled with incremental Gibbs moves.
  • Density estimation: MFMs and DPMs have similar density-estimation behavior, with empirical estimates converging toward the true density and Hellinger distance tending to zero as data increase.The comparison uses data from a three-component Gaussian mixture and is illustrated in Figures 1 and 2.
  • Clustering: MFM priors provide control over component and cluster counts, while DPM priors favor widely varying cluster sizes with many very small clusters.The distinction between a cluster as a data-point group and a component as a mixture distribution is explicit in the paper.
  • Clustering: MFMs and DPMs differ in clustering and component-number behavior: DPMs tend to create tiny extra clusters, whereas MFM cluster counts concentrate at the true value and MFM inference includes a finite component count.In contrast, DPM cluster-count posteriors fail to concentrate, and DPM components are always infinite.

2. Model

The MFM combines a prior on the number of components with symmetric Dirichlet weights and component-specific parameter draws. Its symmetric-weight assumption enables the paper’s simple partition distribution and related representations, while the component-count prior is otherwise lightly restricted.

  • Model specification: The MFM draws the number of components K from an arbitrary probability mass function pK on the positive integers.For consistency of component-number inference, the paper prefers pK(k)>0 for every positive integer k.
  • Model specification: Conditional on K=k, the mixture weights follow a symmetric Dirichlet distribution with a common parameter γ.The single parameter γ does not depend on k.
  • Model specification: Given K=k, latent assignments are independently drawn from the finite weight vector, component parameters independently from H, and observations from fθZj.H is a base measure on Θ, and {fθ: θ∈Θ} is a family of probability densities on X.
  • Assumptions: The symmetric Dirichlet assumption is key to deriving the simple partition distribution and the other resulting MFM properties.The paper motivates symmetry by noting that symmetrizing the weights leaves the distribution of observed data unchanged.

3. Exchangeable partition distribution

The MFM induces a tractable exchangeable partition distribution with a Gibbs-partition form, enabling inference without cluster labels or empty components. Its partition family is consistent under marginalization, and the number of components can be inferred from sampled clusters.

  • Partition distribution: The MFM partition probability is simple enough to compute and depends on the partition through the number and sizes of its parts.Theorem 3.1 gives the partition mass function, with t=|C| and coefficients V_n(t).
  • Exchangeability: The induced partition is exchangeable because its probability is symmetric in the part sizes.The distribution is invariant under permutations of [n].
  • Gibbs structure: The partition law belongs to the Gibbs family, whose utility for inference in finite mixtures with a component-count prior had not previously been explored.This representation connects the MFM to established Gibbs partition theory while highlighting its inferential novelty.
  • Inference representation: Starting from the partition distribution avoids dealing with cluster labels or empty components during inference.The equivalent representation assigns one parameter to each occupied partition part.
  • Component inference: Posterior inference for the number of components K is available from sampled partitions because p(K|T) is easy to compute.The model also supplies conditional independence relations and a recursion for computing V_n(t).
  • Consistency: The partition family is consistent under marginalization, so removing elements from a sample partition yields the corresponding lower-dimensional distribution.This consistency in distribution implies a unique distribution on partitions of the positive integers.

4. Restaurant process, stick-breaking, and random measure representations

MFMs admit restaurant-process and random-discrete-measure formulations analogous to those of DPMs, and a simple stick-breaking construction in a special Poisson case. These representations support direct reuse of DPM inference methods, although incremental Gibbs sampling can mix slowly for large samples.

  • Restaurant process: An MFM restaurant process generates partitions with the MFM partition distribution by seating each new element in an existing cluster or a new cluster.The existing-cluster probability is proportional to |c|+γ, paralleling the Chinese restaurant process.
  • Random measure: The MFM can be represented by a random discrete measure G, from which mixture parameters or observations are sampled.When H is continuous, G has K atoms with probability one; otherwise, fewer than K atoms may occur.
  • Stick-breaking representation: In the special case pK(k)=Poisson(k−1|λ) and γ=1, the MFM weights have a stick-breaking-like representation based on exponential pieces.The construction breaks a unit-length stick using i.i.d. Exponential(λ) pieces until the stick is exhausted.
  • Stick-breaking representation: For that special case, the resulting stick lengths have the same distribution as the MFM mixture weights.This equivalence follows from a standard construction for Poisson processes.

5. Asymptotics

The asymptotic analysis characterizes the MFM’s partition coefficients, component–cluster relationship, and cluster-size distribution. Compared with DPMs, MFMs retain more evenly dispersed cluster sizes rather than concentrating mass near extreme partitions.

  • Asymptotics of V_n(t): V_n(t) asymptotically behaves like the k=t term in its defining series whenever pK(t)>0.Theorem 5.1 provides the corresponding asymptotic expression involving t, γ, and pK(t).
  • Components and clusters: For large n, the MFM’s number of clusters T behaves very similarly to its number of components K under both prior and posterior.Theorem 5.2 formalizes this relationship under positivity conditions on pK(1),…,pK(k).
  • Comparison with DPMs: The cluster-size contrast remains visible in posterior behavior, including the DPM’s tendency toward tiny extra clusters.These extra clusters inflate the sampled number of clusters empirically.
  • Cluster-size distribution: Conditioning on T=t, MFM cluster sizes are approximately shaped like a symmetric t-dimensional Dirichlet distribution.The approximation follows from the asymptotic form of the partition probabilities and the near-equivalence of K and T.
  • Comparison with DPMs: As n grows, DPM cluster-size distributions place mass near simplex corners, whereas MFM cluster sizes remain more evenly dispersed.The DPM pattern corresponds to a few large clusters and many very small ones, unlike the MFM prior tendency toward similarly sized clusters.

6. Inference algorithms

MFMs support partition-based inference algorithms paralleling those used for DPMs, including Gibbs, auxiliary-variable, and split-merge samplers, without reversible jump. However, incremental Gibbs mixing can be slow, especially because MFMs assign little probability to partitions with tiny clusters.

  • DPM-compatible samplers: MFMs inherit DPM-style partition structure, allowing conjugate, non-conjugate, and split-merge MCMC samplers to be applied directly.These properties include an exchangeable partition distribution, restaurant process, random discrete measure formulation, and, in some cases, stick-breaking representation.
  • Gibbs sampling: The MFM Gibbs sampler is a direct adaptation of the DPM Algorithm 3, with V_n(t) replacing the DPM concentration-dependent terms.Computing the required V_n(t) values takes negligible time relative to running the sampler.
  • Non-conjugate inference: For non-conjugate priors, Neal’s Algorithm 8 can be used for MFM inference through the same substitutions used for the conjugate case.The method uses an auxiliary-variable construction when the marginal likelihood cannot be computed easily.
  • Mixing limitations: MFMs may worsen incremental Gibbs mixing because their partition distributions place little probability on states containing tiny clusters.Moving substantial clusters by relocating one element at a time can therefore require long traversals through low-probability regions.
  • Mixing limitations: Split-merge samplers address slow incremental mixing by reassigning many elements in a single move and can be applied directly to MFMs.The paper applies Jain–Neal split-merge samplers together with incremental Gibbs updates in conjugate and non-conjugate settings.

7. Empirical demonstrations

The paper evaluates MFMs on simulated Gaussian mixtures and real galaxy and leukemia datasets using non-conjugate split-merge and Gibbs-based inference. MFMs recover finite-mixture structure while retaining density-estimation and clustering performance comparable to DPMs and published results.

  • Experimental setup: The experiments use Gaussian component densities on simulated and real datasets, although the approach is not limited to Gaussian mixtures.The demonstrations cover density estimation, clustering, and inference for component numbers.
  • 7.1. Simulation example: Simulation results show consistent density estimation for both MFMs and DPMs, while MFM clusterings avoid the small extra clusters common in DPM clusterings.MFMs are also consistent for the number of components, whereas DPMs are not in the reported finite-mixture setting.
  • Galaxy dataset: The galaxy analysis uses the Richardson–Green model and finds an MFM density estimate visually indistinguishable from the published reversible-jump estimate.The estimated MFM posterior on the number of components is also reported as very close to Richardson and Green’s result.
  • Leukemia gene expression dataset: The leukemia experiment analyzes 72 patients using a non-conjugate split-merge sampler with Algorithm 8 and incremental Gibbs updates.The prior is data-dependent, and the sampler includes a hyperprior on a model parameter.
  • Leukemia gene expression dataset: The leukemia posterior concentrates the number of clusters at 3, matching the ALL, MLL, and AML division, while the MFM component posterior shifts slightly rightward.The shift reflects uncertainty about additional components not observed in the data.
  • Leukemia gene expression dataset: The leukemia pairwise probability matrix clearly separates the three groups, with subject 41 assigned toward AML, subject 52 toward MLL, and subject 17 split about evenly between ALL and MLL.The DPM pairwise matrix is reported as indistinguishable from the MFM matrix.

8. Discussion

The discussion presents MFMs as a potentially consistent Bayesian approach to estimating component numbers, while emphasizing that interpretation depends strongly on model assumptions and specification.

  • MFMs provide a Bayesian approach to consistently estimating the number of components, potentially helping assess data heterogeneity.
  • Component-number inference is meaningless when clusters are purely notional and the mixture model is used only for practical clustering.
  • Even when data arise from a mixture, the true component number is difficult to interpret when the component-distribution form is unclear.
  • Broad or weakly specified base measures can affect posteriors on k and t, with large prior variance on normal-mixture means favoring smaller values.The authors recommend varying H and examining posterior sensitivity.
  • Misspecifying the component family can severely affect posteriors on k and t; for Gaussian mixtures outside the true family, they may diverge to infinity as n increases.The authors identify robustness to misspecification as an important problem requiring further study.
  • Clusterings and component-number estimates can still help explore complex, especially high-dimensional, datasets, but are correct only insofar as model assumptions are correct.

Appendix A. Formulas for some posterior quantities

The appendix describes posterior calculations for component counts and predictive densities using posterior partition samples and MFM-specific transformations or normalizations.

  • Posterior samples of C directly estimate t = |C|, while the MFM posterior on k is obtained by transforming the posterior on t using p(k|t).
  • For DPMs, the posterior on the number of components k is always a point mass at infinity.
  • The MFM posterior predictive density can be estimated from partition samples when single-cluster marginals m(xc) are computable.
  • When m(xc) is not readily computable, predictive density estimation uses samples of C, φ and additional parameter draws.
  • A simpler density estimate assumes the new observation joins an existing cluster, approximating the posterior predictive density for sufficiently large n.
  • Posterior predictive calculations can incorporate sampled hyperparameters such as γ for MFMs and α for DPMs.

Appendix B. Proofs

The proofs establish the MFM’s partition, latent-parameter, restaurant-process, random-measure, and asymptotic properties through conjugacy, conditional independence, and limiting arguments.

  • Dirichlet-multinomial conjugacy yields the MFM partition distribution and its normalizing constant Vn(t).
  • The partition C is conditionally independent of K given T, because p(C|t,k) does not depend on k.
  • The observations X are conditionally independent of K given T by combining X ⊥ K | C with C ⊥ K | T.
  • The restaurant-process transition follows by comparing successive partition probabilities and distinguishing singleton creation from assignment to an existing cluster.
  • The MFM admits an equivalent random-measure construction in which latent parameters are generated from a random discrete measure and induce the same joint distribution.
  • Self-consistency enables sequential sampling of partitions, cluster parameters, and observations; with continuous H, distinct parameters make conditioning equivalent to the restaurant-process construction.
  • The asymptotic proofs show that terms with k > t vanish relative to the k = t term, using Stirling’s approximation and dominated convergence.
  • As n grows, the conditional distribution of K given T = t converges to a point mass at k = t, establishing the MFM asymptotic result.
Loading 1502.06241v1…