A Tutorial on Bayesian Nonparametric Models

1 Introduction

• Traditional model selection fits models with different cluster or factor counts, then weighs data fit against a penalty favoring simpler models.The penalty favors models with fewer components or factors.Source: §1 · p. 1
• Bayesian nonparametric models fit a single model whose hidden structure and complexity can grow with the data, rather than specifying complexity in advance.The number of mixture components or factors becomes part of the posterior distribution.Source: §1 · pp. 1–2
• Chinese restaurant process mixtures infer the number of clusters from data and allow previously unseen clusters to appear as new data arrive.Source: §1 · p. 2
• Indian Buffet Process latent factor models infer the number of factors from data and allow that number to grow as new observations arrive.Source: §1 · p. 2
• The tutorial illustrates BNP analysis with published psychological studies, points readers to software, and notes extensions for sequential, grouped, tree, relational, and spatial data.Source: §1 · p. 2

2.1 Finite mixture modeling

• Finite mixtures fix K and generate each observation by sampling a cluster, then its parameterized observation distribution; Bayesian versions place priors on mixing proportions and cluster parameters.A Gaussian mixture is one example, with component parameters serving as means when variances are known.Source: §2.1 · p. 3
• Interpreting clusters as psychological processes requires caution: further experiments are needed to validate their existence and causal relationship to behavior.Source: §2.1 · p. 3

2.2 The Chinese restaurant process

• The CRP assigns customers sequentially, favoring occupied tables in proportion to their occupancy and new tables in proportion to α, while inducing order-invariant partitions.Larger α tends to produce more occupied tables and fewer customers per table.Source: §2.2 · pp. 5–6

2.3 Chinese restaurant process mixture models

• CRP mixtures allow infinitely many possible clusters but only finitely many active ones; α controls the prior expected count, which grows as α log N when α < N/log N.The model associates each cluster with a parameter drawn from G0 and generates observations from the distribution indexed by their assigned cluster.Source: §2.3 · p. 7
• CRP mixture posteriors infer response-time assignments and process counts, retain uncertainty over clusterings, and allow clusters to grow for observations far from the original data.A Gibbs-sampling example captures two primary response-time modes and a small number of left-skewed outliers.Source: §2.3 · p. 7
• A CRP mixture’s posterior predictive distribution can assign a new observation to a previously unseen cluster.Source: §2.3 · p. 7

3 Latent factor models and dimensionality reduction

• The latent factor model represents observations as noisy weighted combinations of factors, with binary masks and continuous weights enabling sparse factor loadings.The mask determines whether a factor is on or off for a dimension; the Bayesian approach places priors on latent factors, masks, and weights and computes their posterior.Source: §3 · p. 9
• IBP factor analysis avoids fixing the number of factors in advance by inferring them from the data and allowing them to grow as new observations arrive.The model uses a binary matrix whose rows correspond to intelligence measures and whose infinitely many columns correspond to latent factors.Source: §3 · pp. 9–10
• In the IBP, each dimension can select multiple latent factors, with dish-selection probability tied to popularity and new dishes drawn from a Poisson(α/N) distribution.The IBP supplies an infinite-capacity prior over latent variables; larger α allows more factors.Source: §3 · p. 10
• Unlike the CRP, which assigns each observation to one component, the IBP allows an observation or dimension to use several components.Latent factor models represent each observation through component activations, providing a lower-dimensional representation when the number of components is smaller than the data dimension.Source: §3 · pp. 7, 9, 10
• Around 4–7 factors describe the 13 reasoning tasks from 234 participants, and the first IBP factor loading is consistent with Spearman’s g.The authors say this does not conclusively rule out a general intelligence factor, but suggests additional factors merit investigation.Source: Figure 6 · p. 12, §3 · p. 12

4 Inference

• For many Bayesian nonparametric models, including those discussed here, the posterior is unavailable in closed form, so inference requires approximation.The tutorial focuses on widely used algorithms rather than a comprehensive treatment of inference methods.Source: §4 · p. 13
• MCMC approximates the posterior by sampling a Markov chain whose equilibrium distribution is that posterior; Gibbs sampling updates hidden variables using their conditional distributions.CRP mixtures are particularly amenable to Gibbs sampling because of exchangeability.Source: §4 · p. 13
• Variational inference searches for the closest member of a simpler distribution family, typically offering faster computation and easier convergence assessment than MCMC without guaranteeing the true posterior.It is guaranteed to recover the true posterior only when that posterior belongs to the chosen family.Source: §4 · p. 14
• MCMC and variational inference provide a data-directed way to search model structures and parameters together, avoiding separate fits for each candidate component count.The tutorial describes this as essential in settings where the search space is difficult to enumerate and explore efficiently.Source: §4 · p. 14

5 Limitations and extensions

• The survey is selective, omitting other BNP ideas including Pitman-Yor processes, gamma processes, Dirichlet diffusion trees, and Kingman’s coalescent.The authors describe the covered models as only part of the broader field.Source: §5 · p. 14

5.1 Hierarchical structure

• Hierarchical Dirichlet processes allow latent components to be shared across individuals, such as topics shared across documents, while accommodating different proportions.The tutorial illustrates this structure with response times from several subjects who share cognitive processes.Source: §5.1 · p. 15

5.2 Time series models

• Infinite HMMs retain the HMM’s dependence on the preceding latent class while allowing infinitely many latent classes; switching linear dynamical systems offer continuous latent states and data-inferred modes.Fox et al. explored nonparametric switching linear dynamical systems using an HDP prior to infer the number of modes.Source: §5.2 · p. 15

5.3 Spatial models

• Spatial BNP models can make the DP base distribution depend on location, and spatially coupled DPs can encode a prior that nearby pixels share an image segment.More general DP dependencies on covariates and extensions to IBP and latent factor models are also noted.Source: §5.3 · p. 15

5.4 Supervised learning

• For supervised learning, Gaussian processes place a prior over nonlinear functions, while CRP-mixture approaches model outputs with conditionally linear means within input-space regions.The tutorial characterizes the Gaussian-process approach as discriminative and the CRP-mixture approach as generative.Source: §5.4 · p. 16

6 Conclusions

• BNP models provide flexible structures that grow and adapt to data, and the tutorial reviews their basics and potential in scientific problems.Source: §6 · p. 16
• BNP models address choosing mixture-component or latent-factor counts, but are not a general solution to model selection.The authors note finite-capacity model comparison and information-theoretic criteria as alternatives in some cases.Source: §6 · p. 16

6.1 Bayesian nonparametric models of cognition

• Cognitive applications include modeling category learning with CRP mixtures and visual feature decomposition with the IBP.Other cited applications include word segmentation, relational theory acquisition, and function learning.Source: §6.1 · pp. 16–17

6.2 Suggestions for further reading

• Hjort et al. (2010) is recommended for applied Bayesian nonparametrics, while Bishop (2006) is suggested for probabilistic statistical machine learning.Source: §6.2 · p. 17

Appendix A: Foundations

• The tutorial uses the CRP and IBP as accessible starting points for BNP models, viewing them as priors over infinite partitions and binary matrices.Source: Appendix A · p. 18
• The appendix turns from these combinatorial representations to the models’ mathematical foundations in constructions of random distributions.Source: Appendix A · p. 18
• The section reviews this random-distribution perspective for both the CRP mixture and the IBP factor model.Source: Appendix A · p. 18

The Dirichlet process

• A Dirichlet process, parameterized by concentration α and base distribution G0, is a distribution over distributions on Θ.A random draw G from it is itself a distribution over Θ.Source: The Dirichlet process · p. 18
• For any measurable partition of Θ, a DP draw assigns masses with a joint Dirichlet distribution parameterized by α times the base-distribution masses.The expected mass in a region is its G0 mass, while larger α concentrates mass more tightly around G0.Source: The Dirichlet process · p. 18
• DP draws are discrete, placing probability on countably infinitely many atoms whose locations are independently drawn from G0.Each atom has an associated probability mass.Source: The Dirichlet process · p. 18
• Marginalizing the DP yields samples with shared values and positive-probability clustering, whose induced partition follows a Chinese restaurant process; the samples are exchangeable.The CRP parameter is α.Source: The Dirichlet process · pp. 18–19

• Marginalizing G makes the Dirichlet process mixture equivalent to a Chinese restaurant process mixture.Source: Dirichlet process mixtures · p. 19
• Gibbs sampling algorithms for Dirichlet process mixtures are based on the Chinese restaurant process representation.Source: Dirichlet process mixtures · p. 19

The stick-breaking construction

• Successive beta-distributed breaks produce infinitely many weights πk, and draws from G0 with these weights yield a DP(α, G0).Sethuraman (1994) showed that the resulting random distribution has this law.Source: The stick-breaking construction · p. 19
• The stick-breaking figure depicts generating π from a unit-length stick and shows the βk draw distribution for different α values.Source: Figure 8 · p. 20

The beta process and Bernoulli process

• In the beta-process construction, weighted atoms have weights between 0 and 1 that need not sum to 1, so the random measure is not a probability measure.The beta process is denoted BP(α, B0); its formal definition is outside the appendix's scope.Source: The beta process and Bernoulli process · p. 21
• Bernoulli-process draws activate subsets of beta-process atoms, yielding an IBP-distributed binary matrix that serves as a latent-factor mask.Factor loadings are generated by drawing atoms and weights, activating subsets, and then generating observations.Source: The beta process and Bernoulli process · p. 21

Stick breaking construction of the beta process

• The beta-process stick-breaking construction reuses the same breakpoints, with weights representing the remaining stick rather than the segment just broken off.Source: Stick breaking construction of the… · p. 21

• The finite mixture assigns classes using multinomial probabilities π and places a symmetric Dirichlet prior with concentration α on π.Source: The infinite limit of finite models · p. 21
• As K→∞, the finite mixture converges to a Dirichlet process mixture.Source: The infinite limit of finite models · p. 22
• As K→∞, the bent-coin factor-mask process converges to the IBP latent factor model.The construction flips a coin with bias wk N times to decide whether each factor is active.Source: The infinite limit of finite models · p. 22

• The appendix presents a table of software packages implementing models from the main text.Source: Appendix B · p. 22