Source-linked AI summary
A simple example of Dirichlet process mixture inconsistency for the number of components
Jeffrey W. Miller, Matthew T. Harrison
TL;DR
DPMs are often used to infer the number of components in finite mixtures, although density consistency does not guarantee component-count consistency. This note analyzes a standard normal DPM and proves that its posterior cluster count is severely inconsistent: for standard normal data, the posterior probability of one cluster goes to 0.
Problem
DPMs are commonly used to infer a finite mixture’s component count from the posterior number of observed-data clusters, but the consistency of this procedure is unclear.
Method
The note studies a standard normal DPM with unit-variance normal components, a standard normal prior on means, and α = 1.
Results
For i.i.d. standard normal data, the posterior probability of one cluster does not converge to 1 and in fact converges to 0.
Takeaways & Limitations
Using the DPM posterior number of clusters to infer the true component count can produce entirely misleading results.
Takeaways & Limitations
The demonstration is restricted to the univariate normal case with unit variance and a standard normal prior on means.
Abstract
from arXiv · showhide
For data assumed to come from a finite mixture with an unknown number of components, it has become common to use Dirichlet process mixtures (DPMs) not only for density estimation, but also for inferences about the number of components. The typical approach is to use the posterior distribution on the number of components occurring so far --- that is, the posterior on the number of clusters in the observed data. However, it turns out that this posterior is not consistent --- it does not converge to the true number of components. In this note, we give an elementary demonstration of this inconsistency in what is perhaps the simplest possible setting: a DPM with normal components of unit variance, applied to data from a "mixture" with one standard normal component. Further, we find that this example exhibits severe inconsistency: instead of going to 1, the posterior probability that there is one cluster goes to 0.
1. Introduction
DPMs can consistently estimate densities without consistently recovering the number of mixture components. This note demonstrates the problem in a standard normal setting and shows that the cluster-count posterior can be severely inconsistent.
- DPM density consistency does not imply consistency for the number of components because negligible-weight superfluous components can preserve a good density estimate.
- Despite having infinitely many components almost surely, DPMs are commonly used to infer finite-mixture component counts from the posterior number of observed-data clusters.
- The note studies a standard normal DPM with unit-variance normal components and a standard normal prior on component means.
- Ignoring tiny clusters may mitigate empirical overestimation, but doing so abandons the DPM model and lacks a clearly corresponding generative model.
- A prior directly placed on the unknown number of components can be consistent for components, proportions, parameters, and density under general conditions.
- The paper proves inconsistency first and then shows, using Hoeffding’s strong law for U-statistics, that the posterior probability of the true component count goes to 0.
2. Setup
The setup defines DPMs through partitions generated by a Chinese restaurant process and a hierarchical mixture model. The analysis specializes this framework to univariate unit-variance normal components with a standard normal prior on means.
- The Chinese restaurant process assigns probabilities to ordered partitions of n observations into t nonempty sets.
- The ordered-partition probability uses concentration parameter α, while uniform permutation of partition parts leaves the prior and posterior on t unchanged.
- A Dirichlet process mixture combines a partition distribution with cluster-specific parameters and component distributions in a hierarchical model.
- The cluster-count prior p_n(t) is obtained by summing partition probabilities over all ordered partitions with t clusters.
- The model distinguishes a component as part of the mixture distribution from a cluster as the data-point indices assigned to a component.
- The special case uses univariate normal components with unit variance and a standard normal prior on component means.
3. Elementary example of inconsistency
The standard normal DPM provides a simple counterexample to consistency for the number of components: with one true component, the posterior probability of one cluster does not converge to 1. The proof uses elementary bounds and shows that the failure extends beyond standard normal data to distributions with finite mean.
- The argument sets α = 1 for simplicity, but the proof extends trivially to every α > 0.
- For i.i.d. N(0, 1) data, the posterior probability p(T_n = 1 | X_1:n) does not converge to 1 under the standard normal DPM with α = 1.This establishes inconsistency when the true number of components is one.
- The proof compares the one-cluster posterior with contributions from two-cluster partitions using bounds derived from the DPM partition probabilities and convexity.The proof specifically uses Equation 2.4, Equation 2.1 with α = 1, and the convexity of x ↦ x^2.
- The result holds almost surely under the i.i.d. standard normal sampling model.
- Finite mean is sufficient for the key law-of-large-numbers step, so the nonconvergence conclusion also applies to i.i.d. data from any distribution with finite mean.
4. Severe inconsistency
The paper strengthens its inconsistency result by proving that, for standard normal data under the standard normal DPM, the posterior probability of one cluster converges to 0. The proof combines divergence of the multi-cluster contribution with boundedness in probability of the one-cluster contribution, using Hoeffding’s strong law for U-statistics.
- Scope: The severe inconsistency conclusion extends easily from α = 1 to any α > 0.The paper states this generalization explicitly.
- The result: Theorem 4.1 establishes that the posterior probability of the true one-component model converges to 0 under standard normal data.This is the paper’s severe inconsistency result.
- Proof strategy: The proof shows that R2(X1:n) diverges in probability while R1(X1:n) remains bounded in probability.These contrasting behaviors are combined to obtain the posterior conclusion.
- Proof strategy: Hoeffding’s strong law of large numbers for U-statistics supplies the key step for proving divergence of R2(X1:n).The argument defines U-statistics from subsets of observations and applies the strong law to them.
- Proof strategy: The one-cluster term is bounded in probability because the normalized sample sum remains O_P(1) under the standard normal distribution.The proof uses the fact that the sample sum is normally distributed for every n.