PaperlayerView exact revision on arXiv

Source-linked AI summary

Nonparametric Bayesian sparse factor models with application to gene expression modeling

David Knowles, Zoubin Ghahramani

arXiv:1011.6293v2stat.APcs.AIstat.ML

TL;DR

The paper addresses sparse factor modeling when both the loading structure and the number of latent factors are unknown. It proposes a fully Bayesian nonparametric model using an Indian Buffet Process prior and evaluates it on synthetic and biological gene-expression data. The reported experiments show that inferred sparsity and latent dimensionality can support predictive performance and interpretability, while the approach has computational limitations in richer models.

  • Problem

    Choosing latent dimensionality and sparse factor connections is difficult, especially because manual dimensionality selection requires a search range and substantial input.

  • Method

    NSFA places a spike-and-slab sparse prior based on the Indian Buffet Process on factor loadings and uses Bayesian sampling to infer factors, connections, coefficients, and hyperparameters.

  • Results

    In synthetic data, the inferred latent-feature count centered near the true K = 16, with E[K] = 16.1 and standard deviation 1.46; experiments also report predictive and interpretability benefits on biological data.

  • Takeaways & Limitations

    IBP-based sparsity provides a framework for inferring latent-factor number while producing more interpretable solutions and, in the reported E. Coli and breast-cancer analyses, improved predictive performance.

  • Takeaways & Limitations

    MCMC becomes slow to mix in all but the simplest models that allow multiple transcription factors to share regulated gene sets.

Abstract

from arXiv · show

A nonparametric Bayesian extension of Factor Analysis (FA) is proposed where observed data $\mathbf{Y}$ is modeled as a linear superposition, $\mathbf{G}$, of a potentially infinite number of hidden factors, $\mathbf{X}$. The Indian Buffet Process (IBP) is used as a prior on $\mathbf{G}$ to incorporate sparsity and to allow the number of latent features to be inferred. The model's utility for modeling gene expression data is investigated using randomly generated data sets based on a known sparse connectivity matrix for E. Coli, and on three biological data sets of increasing complexity.

1. Introduction.

The paper proposes Nonparametric Sparse Factor Analysis (NSFA), combining sparse factor loadings with an Indian Buffet Process prior so latent dimensionality need not be fixed in advance. A fully Bayesian model infers hidden factors, active connections, mixing coefficients, and hyperparameters from the data.

  • 1. Introduction.: Sparsity can improve predictive performance and interpretability by excluding irrelevant factors from individual observed dimensions.The paper motivates loading sparsity using gene regulatory networks, where transcription factors regulate specific gene subsets.
  • 1. Introduction.: Choosing the latent dimensionality K manually requires specifying a search range and significant manual input.The paper contrasts this with methods that infer a posterior over the number of latent dimensions.
  • 1. Introduction.: The IBP defines a distribution over infinite binary matrices, enabling sparse connectivity and straightforward Gibbs-based inference of the appropriate latent dimension.The model can therefore avoid specifying a maximum number of latent dimensions a priori.
  • 1. Introduction.: NSFA extends factor analysis with sparse factor loadings and a potentially unbounded number of latent factors.The model uses the Indian Buffet Process to provide sparsity and infer latent dimensionality.
  • 1. Introduction.: The model uses a spike-and-slab prior for factor loadings, allowing infinitely many candidate sources while only finitely many have nonzero entries.The binary matrix determines whether a loading is active, while active loadings receive Gaussian values.
  • 1. Introduction.: NSFA assumes independent Gaussian noise with diagonal covariance and Gaussian latent-factor priors.Isotropic noise is available as a more restrictive special case.

Start with a finite model.

The finite construction begins with a binary feature-allocation matrix whose columns represent hidden sources and whose entries indicate source activity across observed dimensions. Beta-distributed inclusion probabilities are integrated out using conjugacy before taking the infinite limit.

  • Start with a finite model.: The infinite model is defined by taking the limit of a sequence of finite models as K →∞.The resulting limit is connected to a simple stochastic process.
  • Start with a finite model.: The finite model has D observed dimensions and K hidden sources, with z_dk indicating whether source k contributes to dimension d.Each source is assigned an inclusion probability π_k, and rows are generated independently.
  • Start with a finite model.: The inclusion probabilities π_k use a conjugate Beta(r,s) prior, initially with r = α and s = 1.Here α is the Indian Buffet Process strength parameter.
  • Start with a finite model.: Integrating out the Beta-distributed inclusion probabilities yields the finite prior distribution over the binary matrix Z.This conjugate construction prepares the model for the infinite-dimensional limit.

Take the infinite limit.

Taking the finite construction to the limit produces the Indian Buffet Process, an exchangeable stochastic process for allocating observed dimensions to potentially infinitely many hidden sources. Its strength parameter α controls expected feature usage.

  • Take the infinite limit.: The infinite limit is obtained by ordering nonzero rows of Z and letting K →∞, with K+ denoting the number of active features.The construction also tracks row-pattern counts through the quantities H_D and K_h.
  • Take the infinite limit.: The Indian Buffet Process describes each observed dimension as a customer sampling hidden-source dishes from an effectively infinite buffet.The first customer samples Poisson(α) dishes, while later customers sample existing dishes and then try new ones.
  • Take the infinite limit.: Figure 2 compares IBP draws under α = 4 and α = 8.The figure presents two realizations of the one-parameter IBP for different strength-parameter values.
  • Take the infinite limit.: Exchangeability allows the stochastic-process construction to recover the correct distribution over binary feature-allocation matrices.This property supports sampling-based inference for the feature matrix Z.
  • Take the infinite limit.: Each dimension has a Poisson(α) number of active sources, and the expected number of entries in Z is Dα.These results characterize how the IBP controls feature allocation as the number of observed dimensions grows.

3. Related work.

The paper situates NSFA among Bayesian sparse factor models and sparse PCA methods, emphasizing differences in sparsity priors, generative formulation, computation, and dimensionality selection. Inference uses Gibbs sampling with Metropolis–Hastings steps for new features.

  • 3. Related work.: BFRM is closely related to the finite NSFA model but uses a hierarchical sparsity prior instead of the finite IBP prior.The finite IBP integrates out π_k after assigning π_k a Beta(α/K,1) prior.
  • 3. Related work.: BFRM’s hierarchical sparsity prior is motivated by interpretability but is reported to increase computation and reduce predictive performance.The paper suggests that the uncertainty removed from the posterior may have been informative.
  • 3. Related work.: Sparse PCA methods pursue similar interpretability goals but are not generative models, making regularization and dimensionality selection less direct without cross-validation.The paper compares NSFA experimentally with SPCA after adjusting its regularization constants.
  • 3. Related work.: NSFA infers hidden sources, active features, mixing coefficients, and hyperparameters using Gibbs sampling with Metropolis–Hastings steps for new features.Samples are drawn by successively sampling conditional distributions of model parameters given the remaining Markov-chain state.
  • 3. Related work.: The likelihood assumes independent Gaussian noise with a diagonal noise covariance matrix.The matrix is denoted Ψ in the model description.

Mixture coefficients.

The sampler updates feature activity and mixture coefficients, then proposes new features through a Metropolis–Hastings step. Scheduling these updates improved mixing, especially for adding features.

  • Updating existing features: Gibbs sampling updates each IBP indicator Zdk to determine whether factor k contributes to dimension d.The update integrates out the corresponding loading element gdk when evaluating the likelihood ratio.
  • Updating existing features: Sampling the IBP matrix Z and loading matrix G has computational cost O(NDK).The dominant calculation is performed across N observations, D dimensions, and K factors.
  • Adding new features: κd counts features active only for dimension d and is used when proposing new latent features.Although Z has infinitely many columns, only nonzero columns contribute directly to the likelihood and remain in memory.
  • Adding new features: New features are proposed with a Metropolis–Hastings move over κd and their loading vector g.The proposal integrates out the new latent-factor rows X′ rather than the new loading elements.
  • Adding new features: The proposal modifies the Poisson rate for κd and adds a spike at κd = 1 to avoid rarely proposing new features.The move is accepted with probability min(1,aξ→ξ∗).
  • Sampling schedule: Appropriate sampler scheduling improved mixing, particularly for adding new features.The final scheduling scheme is given in Algorithm 1.

IBP parameters.

The IBP strength parameter can be sampled using a conjugate Gamma prior, while the remaining sampling steps are standard and summarized in the sampler algorithm.

  • IBP parameters: The IBP strength parameter α can be sampled using a conjugate Gamma(e,f) prior.Its posterior update uses the conditional IBP prior as the likelihood term.
  • IBP parameters: The posterior update depends on K+, the number of active sources, and the Dth harmonic number HD.The supplied derivation identifies K+ and HD as components of the update.
  • Sampler implementation: The remaining sampling steps are standard and are included for completeness.Algorithm 1 lists one iteration of the NSFA sampler.

Latent variables.

Latent variables are sampled using a Gaussian conditional whose shared precision structure allows part of the computation to be reused across observations.

  • Latent variables: Because Λ does not depend on n, it is computed and inverted only once per iteration.Sampling X has cost O(K^2 + K^3 + NKD).

Factor precision.

Factor-loading precisions may be shared across factors or estimated separately, with hierarchical priors available to share power across factors and dimensions.

  • Factor precision: Equal factor-loading precisions use a shared λ with a Gamma(c,d) prior and posterior update.This constrains λk = λ across columns of G.
  • Factor precision: Allowing separate precisions assigns each λk a Gamma(c,d) prior and updates them independently from G.The model can additionally sample d to share power across factors.
  • Noise precision: Isotropic noise uses a single inverse variance ψ−1 with a Gamma(a,b) prior.The corresponding posterior update is based on the residual matrix.
  • Noise precision: Independent noise assigns each dimension its own noise variance, which is more appropriate for gene expression data in the authors’ experience.A hyperprior on b produces soft coupling between dimension-specific noise variances.

5. Results.

Across synthetic and biological evaluations, NSFA provided strong sparse-factor recovery and predictive performance while inferring latent dimensionality, though convergence and computational costs depended on proposal settings and data scale.

  • 5.1. Synthetic data.: NSFA and finite SFA performed very well on synthetic reconstruction, outperforming nonsparse and softer-sparsity alternatives.FA overfit as factors increased, while SFA and NSFA achieved the strongest sparse-model performance.
  • 5.1. Synthetic data.: NSFA recovered a posterior mean of 16.1 latent features with fixed α = 1, close to the true K = 16.The posterior standard deviation was 1.46; inferring α increased the mean to 18.3 with standard deviation 2.0.
  • 5.2. Convergence.: Adding mass 0.1 at κ_d = 1 reduced convergence time to around 1500 iterations, whereas prior proposals took around 5000 iterations.Setting λ = 50 increased sampling noise and computational cost because many proposed features were rejected.
  • 5.3. Biological data: E. Coli time-series dataset.: On E. Coli data, sparsity did not improve predictive performance, but ARD or sparse priors alleviated damage from overfitting latent factors.NSFA inferred a median of 4 features, with some probability of inferring 5.
  • 5.4. Breast cancer data set.: On breast cancer data, NSFA avoided time-consuming model selection while matching sparse finite-model performance when an appropriate feature count was chosen.SFA performance was sensitive to K, whereas NSFA remained comparable when the number of features was appropriate.
  • 5.5. Prostate cancer data set.: On prostate cancer data, NSFA achieved better predictive log likelihoods than either finite model for every fixed K and remained feasible with over 10,000 genes.NSFA inferred a median of 12 factors, and 1000 iterations took under 8 hours.
  • 5.5. Prostate cancer data set.: Overall, IBP-based sparsity inferred latent-factor numbers within a theoretical framework and enabled feasible analysis of large gene-expression data sets when factor counts stayed small.Manual selection was time-consuming, while too few factors damaged predictive performance in the breast cancer data set.

6. Discussion.

The authors identify slow MCMC mixing as a limitation when models involve more than the simplest shared regulatory structures.

  • MCMC suffers from slow mixing in all but the simplest models considered.
  • The limitation arises in models where multiple transcription factors may share regulated genes through common motifs.
  • Shared regulatory structure motivates richer models but remains computationally difficult for MCMC.

SUPPLEMENTARY MATERIAL

The supplementary experiment evaluates recovery of sparse connectivity using precision and recall for the active elements of the Z matrix, with results consistent with reconstruction error.

  • Precision and recall are plotted for the synthetic data experiment.
  • The evaluation measures recovery of active elements in the Z matrix after thresholding nonsparse algorithms.
  • The precision-and-recall results are consistent with the reconstruction-error results.
Loading 1011.6293v2…