PaperlayerView exact revision on arXiv

Source-linked AI summary

Variable selection for BART: An application to gene regulation

Justin Bleich, Adam Kapelner, Edward I. George, Shane T. Jensen

arXiv:1310.4887v3stat.ME

TL;DR

The paper addresses variable selection for sparse, high-dimensional problems such as gene regulatory network discovery, where linear models may not capture nonlinear relationships. It adapts BART with permutation-based inference and informed variable-importance priors, finding competitive or improved selection performance across simulations and yeast data. The approach has a computational cost from repeatedly fitting BART to permuted responses.

  • Problem

    Variable selection is challenging in high-dimensional sparse settings, particularly when relationships between predictors and responses are nonlinear.

  • Method

    The paper adapts BART with permutation-based thresholds for variable inclusion proportions and incorporates informed prior information about variable importance.

  • Results

    Across simulations, the BART procedures were competitive for linear relationships and performed better in nonlinear settings; in yeast, they generally selected fewer predictors with similar out-of-sample RMSE and showed better predictive performance per predictor.

  • Takeaways & Limitations

    BART-based variable selection can recover influential predictors while accommodating nonlinear relationships and available biological prior information.

  • Takeaways & Limitations

    The permutation-based approach requires the computational cost of running BART on many permuted response variables and is not presented as formal hypothesis testing.

Abstract

from arXiv · show

We consider the task of discovering gene regulatory networks, which are defined as sets of genes and the corresponding transcription factors which regulate their expression levels. This can be viewed as a variable selection problem, potentially with high dimensionality. Variable selection is especially challenging in high-dimensional settings, where it is difficult to detect subtle individual effects and interactions between predictors. Bayesian Additive Regression Trees [BART, Ann. Appl. Stat. 4 (2010) 266-298] provides a novel nonparametric alternative to parametric regression approaches, such as the lasso or stepwise regression, especially when the number of relevant predictors is sparse relative to the total number of available predictors and the fundamental relationships are nonlinear. We develop a principled permutation-based inferential approach for determining when the effect of a selected predictor is likely to be real. Going further, we adapt the BART procedure to incorporate informed prior information about variable importance. We present simulations demonstrating that our method compares favorably to existing parametric and nonparametric procedures in a variety of data settings. To demonstrate the potential of our approach in a biological context, we apply it to the task of inferring the gene regulatory network in yeast (Saccharomyces cerevisiae). We find that our BART-based procedure is best able to recover the subset of covariates with the largest signal compared to other variable selection methods. The methods developed in this work are readily available in the R package bartMachine.

1. Introduction.

The paper frames sparse, high-dimensional variable selection through gene regulatory network discovery in yeast. It proposes adapting BART to use nonlinear modeling and ChIP-derived prior information.

  • Variable selection identifies the subset of covariates that influence a response variable.
  • The paper focuses on sparse settings where only a small subset of many predictors truly influences a continuous response.
  • Yeast network inference seeks gene–transcription-factor relationships using expression data for approximately 6000 genes and 40 transcription factors.
  • Earlier work used Bayesian linear models with first-order interactions, while biological relationships may also involve nonlinearities and higher-order interactions.
  • ChIP binding data provide prior information, motivating a BART modification that incorporates prior information into variable selection.

2. Techniques for variable selection.

The paper contrasts linear-model variable selection with flexible tree-based methods for complex relationships. Its BART approach uses permutation-based inference and variable inclusion frequency rather than backward stepwise selection or fit improvement.

  • Most classical and Bayesian variable-selection methods assume that the response is a linear function of the predictors.
  • Linear models can mishandle nonlinearities and interactions because researchers must specify them in advance, risking incorrect inclusion or exclusion when the model is misspecified.
  • Tree-based methods such as random forests, boosting, BART, and dynamic trees flexibly approximate complicated response surfaces.
  • Existing tree-based selection methods use splitting variables or predictive-error reductions, including backward stepwise procedures.
  • The proposed BART method uses permutation-based inference rather than backward stepwise selection.
  • Instead of measuring each predictor’s overall fit improvement, the method counts how often that predictor appears in a BART model.

3. Calibrating BART output for variable selection.

BART models nonlinear and interaction effects through ensembles of regression trees, but its variable inclusion proportions can overfit noise in null settings. The paper therefore calibrates selection with permutation-based thresholds that offer different stringency levels and can incorporate prior importance information.

  • BART model: BART models responses as sums of regression trees with homoskedastic normal errors, allowing nonlinearities and interaction effects without a parametric functional form.Each tree partitions predictor space, while the ensemble combines many weak learners; a smaller ensemble is recommended for variable selection.
  • BART model: For variable selection, BART uses a smaller tree ensemble because the primary predicted values do not directly quantify predictor importance.The paper defaults to m = 20 and reports that this medium level aids selection in sparse settings with many predictors.
  • Prior information: BART can incorporate prior information by replacing equal splitting-variable probabilities with informed variable-importance priors.This extension is motivated by applications such as yeast gene-regulatory-network discovery, where prior expectations may distinguish predictor importance.
  • Null calibration: In null data with unrelated predictors, variable inclusion proportions fail to converge to 1/K, reflecting substantial chance capitalization and noise overfitting.For K = 40, some proportions remain above and below 1/40 = 0.025; the pattern persists across response-noise levels and with orthogonalized predictors.
  • Null calibration: The null experiment decomposes variation across data sets, BART runs, and predictors to distinguish data-specific and estimation-related sources of inclusion-proportion variability.Across predictors, within-data-set variation is smaller than between-data-set variation, supporting chance capitalization as a major source.
  • Permutation-based selection: Permutation-based thresholds calibrate variable inclusion proportions without assumptions beyond BART and provide local, global-max, and global-SE selection strategies.The strategies differ in stringency: local selected four predictors, global SE selected two, and global max selected one in the YAL004W example.

4. Simulation evaluation of BART-based variable selection.

Simulations evaluated BART-based variable selection against parametric and nonparametric alternatives under linear and nonlinear settings. BART was competitive in linear settings, stronger in nonlinear settings, and benefited from correctly specified prior information without apparent degradation under misspecification.

  • Simulation design: The evaluation compared BART-based thresholding strategies with stepwise regression, lasso, random forests, and dynamic trees across linear and nonlinear simulations.The BART procedures used variable inclusion proportions and included three thresholding strategies plus cross-validated BART-Best.
  • Evaluation metric: F1 was used because it balances precision and recall more appropriately than out-of-sample RMSE for variable-selection evaluation.RMSE can overweight recall because predictive performance depends heavily on including signal-generating covariates.
  • Linear relationship: In linear simulations, BART-Best was competitive with alternatives, outperforming them in lower-noise sparse settings but trailing lasso and Spike-slab in some higher-noise, less-sparse settings.For p = 500 with p0 = 50, cross-validated BART performed worse than lasso and Spike-slab and was comparable to cross-validated random forests.
  • Threshold selection: More stringent BART thresholds performed better in sparse nonlinear cases, while cross-validation performed about as well as the best individual threshold.The cross-validated threshold version was also reported to perform better than the best individual threshold in the examined sparse cases.
  • Nonlinear relationship: In nonlinear simulations, cross-validated BART-Best had the strongest overall performance and outperformed dynamic trees and cross-validated random forests across the examined situations.BART-Best and cross-validated random forests performed best in low-noise settings, while BART and random forests degraded with higher noise.
  • Informed priors: Correct prior information substantially improved BART-based selection, while incorrect prior information did not degrade performance in the examined settings.The authors attribute this robustness to the Metropolis–Hastings step rejecting splits that substantially reduce posterior value.

5. Application to gene regulation in yeast.

The yeast application uses BART to select transcription factors for each of 6026 genes, incorporating ChIP-derived prior information and tuning its influence. BART-based selection methods generally achieve favorable sparsity–prediction trade-offs, with BART-Best outperforming competing selection procedures on RMSE reduction per predictor.

  • The analysis models each of 6026 genes using expression data from 39 transcription factors across 314 experiments.
  • ChIP experiments provide estimated probabilities that each transcription factor physically binds each gene, which are incorporated as prior information.Predictors with larger prior probabilities receive greater splitting weights in BART.
  • BART-Best tunes prior influence, thresholding strategy, and model selection using training, tuning, and hold-out subsets for each gene.The data are partitioned into 80% training, 10% tuning, and 10% hold-out observations.
  • Across genes, all procedures improve on the null model for most genes, although the null model remains competitive for a minority.This suggests the available transcription factors are not biologically relevant to every gene.
  • BART-G.max selects very few transcription factors, while BART-Best selects fewer than stepwise regression, lasso, RF, and Spike-slab and a number comparable to DT.Figure 7 reports distributions across all 6026 genes; the full BART and OLS models use all predictors.
  • BART-Best generally has larger RMSE reduction per predictor than stepwise regression, lasso, RF, and Spike-slab, while DT is slightly worse on average.BART-Full and OLS perform worse than variable-selection procedures; BART-G.max has the best measure among its 2866 nonempty selections, whereas BART-Best selects at least one factor for 5459 genes.

6. Conclusion.

The paper adapts BART for variable selection using permutation-based thresholds, multiple sparsity assumptions, and cross-validation, while allowing prior information about predictor importance. Simulations and yeast data show competitive or improved selection performance, with future work suggested for variance-reduction metrics.

  • Variable-selection procedures: Permutation procedures establish null distributions for BART variable inclusion proportions and support local, global-maximum, and global-standard-error thresholds.The procedures permute the response, rerun BART, and compare observed inclusion proportions with permutation-derived thresholds.
  • Modeling advantages: BART avoids strong linearity assumptions and provides a principled way to incorporate prior information about relative predictor importance.This addresses settings involving nonlinear relationships and informed variable-importance priors.
  • Empirical findings: In simulations, the procedures are competitive in linear settings, outperform alternatives in nonlinear settings, and improve with correctly specified prior information while remaining apparently robust to misspecification.Comparisons included stepwise regression, lasso regression, random forests, and dynamic trees.
  • Empirical findings: In yeast, the BART procedure generally selects fewer predictors while achieving similar out-of-sample RMSE to lasso and random forests.The application selects transcription factors relevant to individual-gene expression in Saccharomyces cerevisiae.
  • Future directions: Future work could examine variance-reduction metrics within BART as an alternative basis for variable selection.The paper specifically points to a metric explored by Gramacy, Taddy, and Wild (2013).
  • Variable-selection procedures: Cross-validation compares local, global-maximum, and global-standard-error thresholding by test-set L2 error and selects the best method for the full training data.The selected variables are then used to fit BART on the full training set.

SUPPLEMENTARY MATERIAL

Supplementary material provides complete simulation results and additional output for the gene-regulation application.

  • Supplementary material: The supplement contains complete Section 4 simulation results and additional Section 5 gene-regulation outputs.The supplementary material is provided as a PDF identified by DOI 10.1214/14-AOAS755SUPP.
Loading 1310.4887v3…