PaperlayerView exact revision on arXiv

Source-linked AI summary

Inconsistency of Pitman-Yor process mixtures for the number of components

Jeffrey W. Miller, Matthew T. Harrison

arXiv:1309.0024v1math.STstat.MLstat.TH

TL;DR

The paper asks whether the posterior on the number of mixture components concentrates at the true count, a question raised by observed tiny extra clusters. It proves that this posterior is not consistent under broad conditions, including for Dirichlet and Pitman–Yor process mixtures.

  • Problem

    Whether the posterior on the number of clusters concentrates at the true component count remains an unaddressed question, despite simulations showing tiny extra clusters.

    Source: §1.1 · p. 3

  • Method

    The paper proves general theorems for nonparametric mixture models, including Dirichlet and Pitman–Yor process mixtures and broad families of component distributions.

    Source: §1.2 · p. 4

  • Results

    For i.i.d. data from a finite mixture, the posterior probability of the true number of clusters does not converge to 1, except for the stated Pitman–Yor case with σ < 0 and ϑ = N|σ| when t = N.

    Source: §1.2 · p. 4

  • Takeaways & Limitations

    The reported overestimation of component counts through tiny extra clusters is consistent with the theoretical results.

    Source: §1.2 · p. 4, §1.3 · p. 5

    The paper examines fixed σ and ϑ; consistency at t = N is possible in the stated σ < 0 case only when N equals the true component count.

    Source: §1.2 · p. 5

Abstract

from arXiv · show

In many applications, a finite mixture is a natural model, but it can be difficult to choose an appropriate number of components. To circumvent this choice, investigators are increasingly turning to Dirichlet process mixtures (DPMs), and Pitman-Yor process mixtures (PYMs), more generally. While these models may be well-suited for Bayesian density estimation, many investigators are using them for inferences about the number of components, by considering the posterior on the number of components represented in the observed data. We show that this posterior is not consistent --- that is, on data from a finite mixture, it does not concentrate at the true number of components. This result applies to a large class of nonparametric mixtures, including DPMs and PYMs, over a wide variety of families of component distributions, including essentially all discrete families, as well as continuous exponential families satisfying mild regularity conditions (such as multivariate Gaussians).

1. Introduction.

The paper shows that posterior cluster counts from DPMs and PYMs need not recover the component count of a finite mixture. It explains the inconsistency through extra singleton clusters and distinguishes component-count inference from density estimation.

1.1. A motivating example.

  • Investigators use posterior observed-cluster counts to infer finite-mixture component counts, although consistency of that procedure was an open question.The motivating example is impala population structure, where the distinction matters for conservation-related studies.Source: §1.1 · pp. 1, 3

1.2. Overview of results.

  • Under fairly general conditions, DPM and PYM posteriors on the number of clusters do not concentrate at any finite true component count.The results cover broad partition distributions and component families; i.i.d. sampling from a finite mixture is stronger than required.Source: §1.2 · p. 4
  • The stated corollary includes discrete families with shared point masses, such as Poisson, Geometric, Negative Binomial, Binomial, and Multinomial models.It also allows continuous base measures for the listed discrete case.Source: §1.2 · p. 4
  • The analysis fixes σ and ϑ; cases where ϑ varies with sample size or has a prior are conjectured to remain inconsistent, but are not examined.For σ < 0, the exception t = N can be consistent only when N equals the true component count.Source: §1.2 · p. 5

1.3. Discussion / related work.

  • DPM density-estimation guarantees do not imply component-count consistency, since mixtures with extra components can approximate the same density.The paper notes that consistency for component counts can instead be obtained by placing a prior on the number of components, though misspecification may strongly affect estimates.Source: §1.3 · pp. 5–6

1.4. Intuition for the result.

  • Conditional on a fixed cluster count, partition sizes concentrate near configurations with small clusters, making singleton splits into an additional cluster comparatively plausible.When the likelihood is also of the same order, the posterior need not make the count with one extra cluster much less probable.Source: §1.4 · pp. 6–7

1.5. Organization of the paper.

  • The paper defines Gibbs partition mixtures, proves a general inconsistency theorem, applies it to bounded-density and regular exponential-family cases, and develops the needed marginal inequalities.The later sections establish the exponential-family inequality used in the application.Source: §1.5 · p. 7

2. Model distribution.

The model generalizes Pitman–Yor and Dirichlet process mixtures through distributions on ordered partitions, then assigns component parameters and observations through a hierarchical mixture. It tracks the number of observed-data clusters, whose posterior is the focus of the paper.

2. Model distribution.

  • The model extends Pitman–Yor mixtures to Gibbs partition mixtures, with Dirichlet and Pitman–Yor process mixtures as special cases.The model combines a partition distribution, a prior on component parameters, and component-specific data distributions.Source: §2 · p. 7, §2.2 · p. 9

2.1. Gibbs partitions.

  • A Gibbs partition assigns probabilities to ordered partitions of the indices into nonempty parts, with the number of parts determining the cluster count.Uniformly permuting partition parts yields the ordered-partition distribution without changing the distribution of the number of parts.Source: §2.1 · pp. 7–8

2.2. Gibbs partition mixtures.

  • In the hierarchical mixture, each part indexes observations assigned to a component, and T_n records the number of clusters among n observations.The posterior of interest is the probability of T_n=t given the observed data.Source: §2.2 · p. 9

3. Inconsistency theorem.

Under Conditions 3.1 and 3.2, the theorem rules out posterior concentration at any fixed cluster count in Gibbs partition mixtures. For data generated i.i.d. from a finite mixture with t components, the posterior on the number of clusters is almost surely inconsistent for t.

3. Inconsistency theorem.

  • Condition 3.1 concerns partition distributions, whereas Condition 3.2 concerns the data distribution and the single-cluster marginals.The theorem's assumptions therefore separate properties of the partition model from properties of the data and component model.Source: §3 · pp. 10–11
  • For Pitman–Yor processes, Condition 3.1 holds for every t when σ∈[0,1) and ϑ∈(−σ,∞), and for all t except N when σ<0 and ϑ=N|σ|.Condition 3.1 bounds the relevant partition-distribution quantities as n grows.Source: §3 · pp. 10–11
  • For a Gibbs partition mixture model satisfying Conditions 3.1 and 3.2, p(T_n=t | X_1:n) < 1 with probability 1; if observations are i.i.d. from a t-component mixture, the posterior of T_n is not consistent for t.Source: §3 · p. 11
  • The proof bounds the joint density contributions for t and t+1 clusters, using the combinatorial step of moving one observation into a new singleton part.The stated bound is p(x1:n,T_n=t)+p(x1:n,T_n=t+1)≤C under the lemma's conditions.Source: §3 · pp. 11–13

4. Application to discrete or bounded cases.

Theorem 4.1 yields Condition 3.2 under boundedness conditions, implying inconsistency for broad discrete and bounded cases. These include finite sample spaces with positive marginal mass, families sharing a point mass with the data, and certain continuous families.

4. Application to discrete or bounded cases.

  • Theorem 4.1 implies inconsistency for a large class of Pitman–Yor mixture models, including essentially all discrete cases and some continuous cases.The theorem applies when its conditions hold; the section gives examples of those conditions and their scope.Source: §4 · p. 13
  • When m(x) > 0 throughout a finite sample space, Theorem 4.1’s conditions hold regardless of the distribution of the observations.For finite X, mixture identifiability may hold only up to a maximum component count, though consistency might still be possible within that range; the result nevertheless shows PYMs are inconsistent.Source: §4 · p. 14
  • A shared point mass in the data distribution, base measure, and single-cluster marginal suffices for Theorem 4.1’s conditions.Specifically, there must be an x0 with P(x0) > 0, λ(x0) > 0, and m(x0) > 0.Source: §4 · p. 14
  • For countable discrete families, the result applies when some x0 has both P(x0) > 0 and m(x0) > 0; otherwise, model and data distributions are mutually singular.Examples include Poisson, Geometric, Negative Binomial, and power-series distributions, provided the data are i.i.d.Source: §4 · p. 14
  • The bounded-density conditions cover continuous densities bounded on a non-null compact set, but Theorem 4.1 does not cover some important cases, including multivariate Gaussians with unrestricted covariance.The paper addresses many exponential-family cases separately in the remainder.Source: §4 · pp. 14–15

5. Exponential families and conjugate priors.

The section sets up exponential families and their conjugate priors, then defines the well-behaved class used in later results. Its regularity conditions hold for many commonly used families, including multivariate Gaussians, Gamma, Poisson, Exponential, and Geometric.

5.1. Exponential families.

  • The paper represents an exponential family by densities pθ with respect to a sigma-finite Borel measure and assumes mild regularity conditions.The conditions require the family to be full, nonempty, regular, and identifiable.Source: §5.1 · p. 15
  • Many commonly used exponential families satisfy these conditions, including multivariate Gaussian, Gamma, Poisson, Exponential, and Geometric families.The Inverse Gaussian is a stated exception because its parameter space is not open.Source: §5.1 · p. 15

5.2. Conjugate priors.

  • A well-behaved exponential family with conjugate prior is defined by the stated exponential-family form, Conditions 5.1, and (ξ, ν) ∈ Ξ.The section notes that the specified prior family is conjugate for {Pθ}.Source: §5.2 · p. 16

6. Application to exponential families.

For well-behaved exponential families with conjugate priors, Theorem 6.2 establishes Condition 3.2 for data satisfying Conditions 6.1, implying inconsistency in many finite-mixture cases. Proposition 6.3 supplies conditions under which finite mixtures satisfy those data conditions.

6. Application to exponential families.

  • Theorem 6.2 establishes Condition 3.2 for every t when data are i.i.d. from a distribution satisfying Conditions 6.1 and the model uses a well-behaved exponential family with conjugate prior.The theorem follows from a result whose proof occupies much of the remainder of the paper.Source: §6 · p. 16
  • The result implies inconsistency when the data come from a finite mixture from the same family and satisfy Conditions 6.1.Proposition 6.3 provides conditions ensuring that such finite-mixture data satisfy those conditions.Source: §6 · pp. 16–17
  • The paper says these results, combined with Theorem 4.1, cover a fairly comprehensive range of cases.The section’s contribution is to establish inconsistency for many well-behaved exponential families with conjugate priors.Source: §6 · pp. 16–17
  • Proposition 6.3’s sufficient conditions include a continuous family on an open connected X with real-analytic sufficient statistics.The proposition also assumes Conditions 5.1 and the finite-mixture setup described in its statement.Source: §6 · p. 17

7. Marginal inequalities.

This section develops uniform bounds on exponential-family marginal densities and a splitting inequality. The bounds use Laplace approximation beyond the i.i.d. setting, for observations whose sufficient statistics are not too extreme.

7. Marginal inequalities.

  • The Laplace approximation is used beyond the i.i.d. case, for samples whose sufficient statistics are not too extreme.The text contrasts this with the commonplace i.i.d. application to m(X1:n).Source: §7 · p. 18
  • Proposition 7.1 bounds the ratio m(x1:n)/em(x1:n) above and below uniformly when the sample’s sufficient-statistic mean lies in a compact subset of the moment space.The approximation em(x1:n) is intended only up to a multiplicative constant.Source: §7 · p. 18
  • Proposition 7.2 gives a splitting inequality for nonempty subsets A and B when the full-sample sufficient-statistic mean lies in a compact set.For such samples, the marginal of the full sample is bounded by a constant times the product of the marginals for the two subsets.Source: §7 · p. 20
  • The splitting bound is derived using convexity of the Legendre transform and uniform marginal bounds over a compact set.The proof applies the bounds to the full sample and its two subsets.Source: §7 · pp. 20–21

8. Marginal inequality for subsets of the data.

This section develops compact-set capture results for exponential-family sufficient statistics and posterior means, then uses them to establish a subset-wise marginal bound. The supporting results show that sufficiently large subsets contain many observations whose posterior means lie in a common compact set.

8. Marginal inequality for subsets of the data.

  • The Capture lemma states that, under its conditions on Q and an open convex V, a compact subset U of V captures the required fraction of iid observations asymptotically.The supplied statement includes the compact-set conclusion and conditions involving hyperplanes, but does not provide the full conditions or displayed conclusion.Source: §8 · p. 21
  • For iid Z1, Z2, ... ∈Rk, Proposition 8.2 states an almost-sure limit of 1 as n→∞ under β∈(0,1] and U⊂Rk, with Yj=I(Zj∈U).Source: §8 · p. 21
  • For any β in (0, 1], Proposition 8.3 supplies a compact U in M that captures asymptotically all but a β/2 fraction of the sampled posterior means.The result assumes a well-behaved exponential family with conjugate prior and P(s(X) ∈ M) = 1.Source: §8 · p. 22
  • For every sufficiently large subset J, Lemma 8.4 guarantees a subset K of at least half its size whose observations have posterior means in a common compact set.The compact set and positive constant are chosen under the stated well-behaved exponential-family, conjugate-prior, and sampling assumptions.Source: §8 · p. 22

APPENDIX A

Appendix A bounds the posterior probability of a fixed component count under the stated conditions. Its theorem proof concludes that this probability cannot converge to one.

APPENDIX A

  • Proposition 3.3 bounds the relevant weight ratios asymptotically across the stated σ and ϑ cases, with an exception at t = N in case (B).In case (B), the passage states the bound when t < N and separately describes ratios for N < t < n.Source: Appendix A · p. 23
  • Theorem 3.4 concludes that the posterior probability of any fixed t components has limit superior strictly below one under Conditions 3.1 and 3.2.The proof assumes the stated lower-bound condition on ϕ_t(x_1:n, c), which it says occurs with probability 1.Source: Appendix A · p. 23

APPENDIX B: EXPONENTIAL FAMILY PROPERTIES

Appendix B collects properties of exponential families under Conditions 5.1. It also establishes finiteness of the single-cluster marginal under the stated assumptions.

APPENDIX B: EXPONENTIAL FAMILY PROPERTIES

  • Under Conditions 5.1, the log-partition function κ is smooth and strictly convex, its gradient maps Θ smoothly and bijectively onto M, and M is open and convex.The appendix also identifies κ′(θ) as the expectation of s(X) and κ′′(θ) as its covariance under Pθ.Source: Appendix B · p. 24
  • For the conjugate family, updating parameters with observed sufficient statistics preserves the parameter set, and the resulting family is conjugate for the component distributions.The appendix further states that the single-cluster marginal m(x_1:n) is finite under the stated continuity and measure assumptions.Source: Appendix B · p. 25
  • The appendix notes that Ξ may be a strict subset of the parameters on which πξ,ν can be defined.The displayed properties are collected as well-known exponential-family results, with references including Hoffmann-Jørgensen (1994).Source: Appendix B · pp. 24–25

APPENDIX C: BOUNDS ON THE LAPLACE APPROXIMATION

Appendix C gives upper and lower bounds for a Laplace approximation under local curvature and separation conditions. A corollary provides uniform bounds when eigenvalues and other quantities satisfy fixed constraints.

APPENDIX C: BOUNDS ON THE LAPLACE APPROXIMATION

  • The authors describe these bounds as a multivariate version of De Bruijn’s bounds and say the precise formulation needed for their proof was not found in the literature.They place some technical proofs in a supplementary document excluded from the main paper for space.Source: Appendix C · p. 25, Supplementary Material: Supplement to… · p. 26, Brown University · p. 29
  • Lemma C.1 bounds the approximation above and below when f has a stationary point x0, its Hessian is bounded between positive-definite matrices nearby, and f is separated outside that neighborhood.The lemma applies for 0 < s ≤ t under the stated ball-containment and positive separation conditions.Source: Appendix C · pp. 25–26
  • Corollary C.2 bounds the Laplace-approximation quantity uniformly for all t ≥ 1 when its stated constraints on E, f, x0, A, and B hold.The bounds require lower and upper eigenvalue constraints on A and B and an upper bound on the specified function value.Source: Appendix C · p. 26

S1. Miscellaneous proofs.

The supplementary section collects technical proofs and states a Laplace-approximation corollary. Its visible proof excerpts include an argument that certain analytic zero sets have measure zero under stated assumptions.

Other proof fragments

  • The remaining excerpts provide fragments of proofs for Proposition 6.3 and Lemma C.1, without enough detail here to state their conclusions.The text references inequalities, an integral outside Bε(x0), and Taylor’s theorem.Source: S1. Miscellaneous proofs · pp. 29–30

Analytic zero-set argument

  • For an open connected domain and real-analytic s, the zero set of the associated hyperplane function has Lebesgue measure zero unless that function is identically zero.The proof rules out the identically-zero case using properties of Pθ, then uses absolute continuity to conclude P assigns the zero set probability zero.Source: S1. Miscellaneous proofs · pp. 29–30

Laplace approximation

  • The corollary gives the asymptotic form of the Laplace approximation under smoothness, curvature, uniqueness, and separation conditions.The excerpt specifies a C2 function with a critical point whose second derivative is positive definite, and requires that point to be the unique minimizer; the remaining condition is truncated.Source: S1. Miscellaneous proofs · p. 30

S2. Capture lemma.

The capture lemma constructs a compact set inside an open convex region using finitely many halfspaces. Its proof reduces the construction to one-dimensional tail control and geometric covering arguments.

Capture lemma

  • For iid samples from P, the capture lemma asserts that, for any β in (0, 1], a compact subset U of V can be chosen with the stated asymptotic capture property.The displayed conclusion is truncated; the proof excerpt constructs U from halfspaces whose sample capture indicators converge to one.Source: S2. Capture lemma · p. 31

Capture lemma construction

  • For each direction, a halfspace containing the origin is chosen so its boundary meets V along the ray and its sample capture indicator converges to one.Intersecting finitely many such halfspaces yields a compact set contained in V.Source: S2. Capture lemma · pp. 31, 33

Proposition S2.2

  • For a real-valued iid sample with finite first absolute moment, some threshold b below c makes Iβ(X1:n, (−∞, b]) converge almost surely to one.The proof chooses a cutoff a with P(X>a)<β and a larger threshold b below c, then uses the largest observations to establish the capture property.Source: S2. Capture lemma · pp. 31–32

Proposition S2.5

  • The finite intersection is compact because it is closed and bounded; unboundedness would force a ray to remain inside it, contradicting one of its halfspace bounds.The proof uses compactness of the unit sphere to extract a limiting direction from an unbounded sequence.Source: S2. Capture lemma · p. 34
Loading 1309.0024v1…