Source-linked AI summary
PReMiuM: An R Package for Profile Regression Mixture Models using Dirichlet Processes
Silvia Liverani, David I. Hastie, Lamiae Azizi, Michail Papathomas, Sylvia Richardson
TL;DR
PReMiuM addresses Bayesian clustering of responses and covariates while also identifying covariates that drive mixture components. It implements Dirichlet process mixture samplers and related tools, with stable simulated results and variable-selection estimates consistent with the data.
Problem
PReMiuM addresses the need to link responses and covariates through mixture-component membership and identify which covariates actively drive those components.
Method
PReMiuM implements Bayesian Dirichlet process mixture clustering with multiple samplers, retrospective and auxiliary-variable sampling, post-processing, and variable selection.
Results
The algorithm consistently sampled variable-selection values ρp in accordance with the simulated data, while independent chains showed virtually identical results in simulated runs.
Takeaways & Limitations
PReMiuM provides an R-based tutorial and toolkit for Dirichlet process clustering, prediction, missing covariates, convergence assessment, post-processing, and variable selection.
Takeaways & Limitations
Convergence can remain problematic because cluster-specific parameters change in number and labels, so users must assess it with diagnostic tools.
Abstract
from arXiv · showhide
PReMiuM is a recently developed R package for Bayesian clustering using a Dirichlet process mixture model. This model is an alternative to regression models, non-parametrically linking a response vector to covariate data through cluster membership. The package allows Bernoulli, Binomial, Poisson, Normal and categorical response, as well as Normal and discrete covariates. Additionally, predictions may be made for the response, and missing values for the covariates are handled. Several samplers and label switching moves are implemented along with diagnostic tools to assess convergence. A number of R functions for post-processing of the output are also provided. In addition to fitting mixtures, it may additionally be of interest to determine which covariates actively drive the mixture components. This is implemented in the package as variable selection.
1. Introduction
PReMiuM implements profile regression as Bayesian clustering with a Dirichlet process mixture model, linking responses to covariates through cluster membership. The package combines multiple data-modeling options with samplers, diagnostics, post-processing, and variable selection.
- Motivation and package contribution: PReMiuM implements profile regression as Bayesian clustering using a Dirichlet process mixture model.Profile regression links response vectors to covariate data through cluster membership.
- Motivation and package contribution: The package supports binary, categorical, count, and continuous responses alongside continuous and discrete covariates.
- Motivation and package contribution: PReMiuM provides multiple samplers, label-switching moves, convergence diagnostics, and R functions for post-processing model output.
- Motivation and package contribution: The implementation includes predictions for responses and handling of missing covariate values.
- Motivation and package contribution: Variable selection identifies covariates that actively drive mixture components or share characteristics across components.
- Dirichlet process background: The package implements Dirichlet and Poisson-Dirichlet processes, while the proposed methods also hold for more general stick-breaking formulations.
2. Sampling the Dirichlet process mixture model
PReMiuM samples Dirichlet process mixture models using latent allocations, stick-breaking weights, and several finite or infinite-dimensional MCMC strategies. Slice and retrospective sampling jointly update relevant components while avoiding explicit updates over infinitely many parameters.
- Model representation: The DPMM represents observations through component-specific densities, latent allocations, stick-breaking weights, and global parameters.
- MCMC samplers: PReMiuM implements truncated, slice dependent, and slice independent samplers for Dirichlet process mixture models.
- MCMC samplers: Truncated sampling can produce a posterior different from the full model when the truncation level C is insufficiently large.
- Slice sampling: Auxiliary variables and retrospective sampling yield a Gibbs sampler whose marginal samples follow the desired DPMM posterior.
- Slice sampling: Slice samplers support label-switching moves and update only finitely many relevant components despite the infinite mixture representation.
- Slice sampling: At each sweep, active, potential, and inactive component sets partition the positive integers, with observations allocated only to active components.
3. Example models
PReMiuM applies its DPMM framework to Gaussian and categorical covariates, while profile regression jointly models covariates, responses, and optional global fixed effects. The package supports several response models and corresponding Bayesian updating schemes.
- Covariate models: The package provides Gaussian and discrete categorical mixture models for covariate data.Gaussian clusters use means and covariance matrices; categorical clusters use category-probability vectors.
- Covariate models: Gaussian mixture parameters receive Normal and inverse-Wishart priors that permit Gibbs updates for active and potential clusters.
- Covariate models: Categorical mixture probabilities receive conjugate Dirichlet priors and can be updated directly with Gibbs sampling.
- Profile regression: Profile regression jointly models response and covariate data, allowing covariate effects on responses to depend on cluster membership.
- Profile regression: Optional fixed effects are constrained to have global rather than cluster-specific effects on the response.
- Response models: Implemented response models include binary, categorical, Binomial, and other response choices whose likelihood depends on the selected model.
Count response modelled as Poisson
PReMiuM supports Poisson and overdispersed response models alongside continuous-response models, while also providing discrete and soft variable-selection formulations for covariates.
- The Poisson model provides an alternative to the Binomial model for count-type responses with an individual-specific expected offset E_i.
- Extra response variation is modelled by introducing latent λ_i terms and an additional variance-related parameter σ^2_ε.
- The precision τ_ε = 1/σ^2_ε receives a Gamma prior, enabling a simple Gibbs update, while latent λ_i values are updated at every MCMC sweep.
- Continuous responses are supported through cluster-specific parameters and a variance parameter σ^2_Y with a conjugate Gamma prior on τ_Y = 1/σ^2_Y.
- Variable selection: The soft-selection formulation allows only global variable selection and is described as less likely to encounter mixing problems than the more elaborate first approach.
4. Predictions
PReMiuM predicts responses for user-defined pseudo-profiles by allocating them to fitted clusters, while leaving model fitting unchanged and accommodating missing predictive covariates.
- Users specify predictive scenarios, or pseudo-profiles, to examine covariate effects and compare the risk associated with different covariate profiles.
- Predictive subjects do not affect the likelihood, clustering, or parameters, and missing values in predictive scenarios are ignored.
- Predictions can use simple cluster allocations or Rao-Blackwellised estimates based on allocation probabilities.
- Prediction densities over MCMC sweeps estimate pseudo-profile effects and support comparisons among covariate profiles.
- When fixed effects are missing, their mean or zero-category effect is used; missing offsets or trial counts are set to 1.
5. Postprocessing of the MCMC output
PReMiuM post-processes MCMC output by selecting representative partitions and quantifying uncertainty in cluster risks and profiles.
- Posterior output supports learning about partition uncertainty and displaying a representative partition.
- Label switching prevents direct assignment by maximum average posterior cluster probability because the number of clusters can vary across MCMC samples.
- PReMiuM computes posterior similarity matrices and provides Binder-loss and PAM-based procedures for selecting an optimal partition.
- Given an optimal partition, the package examines MCMC output to assess whether individuals are consistently clustered similarly across iterations.
- For each cluster, averaging baseline risks across its members at every iteration yields an empirical baseline-risk distribution whose credible intervals narrow with consistent clustering.
- The same distributional approach can be applied to cluster parameters for other response and covariate types.
6. Mixing of the MCMC algorithm
PReMiuM addresses label switching with specialized moves and adds diagnostics, but convergence remains difficult to assess and optimal partitions do not directly quantify clustering uncertainty.
- Because the likelihood is invariant to cluster-label order but the stick-breaking prior is not, label-switching moves are needed for adequate mixing across orderings.
- The package implements two label-switching moves from Papaspiliopoulos and Roberts and a third from Hastie et al.
- Convergence may remain problematic because fixed effects, changing cluster parameters, and the concentration parameter α are unreliable indicators of overall convergence.
- The marginal model posterior p(Z|D) is implemented as a diagnostic for comparing MCMC runs with different initialisations.
- Optimal partitions should be interpreted cautiously because no effective method is available to directly quantify or visualise clustering uncertainty.
7. Software
PReMiuM is distributed as an R package implementing the paper’s DPMM algorithm, with configurable hyperparameters and prediction methods.
- The DPMM algorithm is available as an R package primarily written in C++ and R.
- The sampler implements the algorithm described in the paper, while further model extensions remain in progress.
- Users can customise model hyperparameters through name-value pairs, with defaults applied when values are unspecified.
- Predictions can use random allocations or Rao-Blackwellisation based on allocation probabilities.
8. Examples
The examples demonstrate PReMiuM’s clustering, prediction, variable-selection, and convergence-diagnostic workflows. Simulations show stable chains, predictions consistent with the generated data, identification of irrelevant covariates, and diagnostic agreement across initialisations, while convergence is not guaranteed.
- 8.1. Simulated example: The simulated dataset contains 1,000 subjects in five balanced groups, ten binary covariates, and a binary response, with only the first eight covariates supporting clustering.
- 8.1. Simulated example: Independent chains initialized with 20 groups produced virtually identical results, although convergence remains complex to ensure.
- 8.1. Simulated example: The representative clustering contains five clusters whose posterior response and covariate probabilities are visualized in Figure 1.
- 8.2. Predictions: Missing covariate values are ignored in prediction, with their marginal effect interpreted as a population-average effect; predictions match the simulated data.
- 8.3. Variable selection: Variable selection identifies covariates 9 and 10 as lacking clustering support and excludes them from affecting the clustering.
- 8.4. Assessing convergence: No method assures that MCMC chains have converged, so diagnostics can only investigate evidence against convergence.
- 8.4. Assessing convergence: Convergence diagnostics include global-parameter traces, autocorrelation plots using coda, and marginal model posterior comparisons across initialisations.
- 8.4. Assessing convergence: For the simulated dataset, four runs with different initial cluster counts reached subsets of model space with equivalent marginal model posterior, which does not imply convergence.
Conclusions
PReMiuM provides tutorial-oriented tools for Dirichlet process clustering, Bayesian sampling, post-processing, and diagnostics, with future development aimed at complex, high-dimensional data and broader data types.
- PReMiuM illustrates Dirichlet process clustering through a sampler and post-processing tools designed to demonstrate the package’s utility.The authors describe the paper as a tutorial and note that the package objects support additional uses beyond those detailed.
- The authors plan to extend PReMiuM for complex and high-dimensional datasets and increase flexibility across analyzable data types.
A. Appendices
The appendices detail the blocked MCMC sampler for the Dirichlet process mixture model, including joint updates, label switching, parameter updates, and finite conditional sampling after marginalization.
- Blocked sampler steps: The appendix organizes the sampler into sequential steps that update active components, latent variables, global parameters, and allocations.The blocking strategy is explained through Steps A–G, with detailed updates for mixture weights, component parameters, likelihood parameters, and allocations.
- Label switching: Label-switching moves jointly update active weights, component parameters, and allocations, with Gibbs-form updates equivalent to Metropolis-Hastings moves when acceptance probability is 1.
- Latent variables and allocations: The sampler updates latent slice variables independently from uniform full conditionals and updates allocations using finite multinomial weights for clusters with posterior mass.The allocation update depends only on active and prospective components permitted by the slice variables.
- Global and component parameters: Global concentration and component-weight updates use conditional factorization, adaptive Metropolis-within-Gibbs sampling, and checks that determine the active component set.Empty prospective and inactive components are handled through prior sampling or omitted from the sweep when they do not affect the likelihood.
- Variable selection: The appendix gives variable-selection matrix definitions and posterior conditional updates for continuous-covariate cluster means.The matrix constructions distinguish covariates selected under the two variable-selection methods.