Source-linked AI summary
BART: Bayesian additive regression trees
Hugh A. Chipman, Edward I. George, Robert E. McCulloch
TL;DR
BART addresses the challenge of modeling regression functions with both additive and interaction effects while retaining uncertainty quantification. It combines regularized weak trees with Bayesian backfitting MCMC, and performs favorably across diverse datasets while providing reliable posterior estimates and marginal effects.
Problem
Flexible regression models need to accommodate both additive effects and interactions more naturally than low-dimensional smoothers or single trees.
Method
BART models the regression function as a regularized sum of weak trees and uses backfitting MCMC to obtain posterior inference, uncertainty intervals, marginal effects, and variable-usage frequencies.
Results
Across diverse datasets and simulations, BART compared favorably with boosting, lasso, MARS, neural nets, and random forests, while yielding reliable posterior mean, interval, and marginal-effect estimates.
Takeaways & Limitations
BART provides an effective, robust, and easy-to-use tool for predictive modeling, posterior inference, and model-free identification of important features.
Takeaways & Limitations
Spurious splits can inflate variable-usage frequencies, making important variables harder to distinguish.
Abstract
from arXiv · showhide
We develop a Bayesian "sum-of-trees" model where each tree is constrained by a regularization prior to be a weak learner, and fitting and inference are accomplished via an iterative Bayesian backfitting MCMC algorithm that generates samples from a posterior. Effectively, BART is a nonparametric Bayesian regression approach which uses dimensionally adaptive random basis elements. Motivated by ensemble methods in general, and boosting algorithms in particular, BART is defined by a statistical model: a prior and a likelihood. This approach enables full posterior inference including point and interval estimates of the unknown regression function as well as the marginal effects of potential predictors. By keeping track of predictor inclusion frequencies, BART can also be used for model-free variable selection. BART's many features are illustrated with a bake-off against competing methods on 42 different data sets, with a simulation experiment and on a drug discovery classification problem.
1. Introduction.
BART models f(x) = E(Y |x) with a regularized sum of regression trees whose weak effects form a dimensionally adaptive random basis. Bayesian backfitting MCMC yields posterior inference, including point estimates and uncertainty intervals for the regression function.
- Model: BART models f(x) = E(Y |x) as a sum of regression trees, with a prior that regularizes individual tree effects into weak learners.The regularized tree effects act as a dimensionally adaptive random basis.
- Motivation: The sum-of-trees structure can incorporate interaction effects more naturally than low-dimensional smoothers and additive effects more easily than a single tree.
- Fitting: A tailored Bayesian backfitting MCMC algorithm iteratively fits successive residuals using a fixed number of trees, rather than sequential boosting or a tree-weakening procedure.BART weakens trees through its prior and performs iterative fitting through Bayesian backfitting.
- Inference: Posterior draws provide point estimates by averaging evaluated sum-of-trees models and pointwise uncertainty intervals from their corresponding quantiles.
- Implementation: Open-source BART software is provided as a standalone package and through an R interface, with documentation and examples.The implementation is available as the BayesTree library in R.
2. The BART model.
BART models the regression function as a sum of binary regression trees, with each tree contributing a partial effect. A regularization prior keeps individual trees weak while data-informed hyperparameter calibration supports flexible yet stable function approximation.
- Sum-of-trees representation: BART represents E(Y |x) as the sum of terminal-node parameters assigned by multiple binary regression trees, so each tree contributes only part of the regression function.With multiple trees, components can represent main effects or interactions depending on which predictors their splitting rules use.
- Prior regularization: The regularization prior limits individual tree effects because otherwise large components could overwhelm the additive representation and reduce its approximation and computational advantages.Increasing the number of trees increases representational flexibility but also increases the number of parameters, making regularization important.
- Tree prior: Tree-structure priors favor small components; with α = 0.95 and β = 2, trees with 1, 2, 3, 4, and ≥5 terminal nodes receive probabilities 0.05, 0.55, 0.28, 0.09, and 0.03.Splitting variables and available splitting values use uniform default priors.
- Parameter prior calibration: The normal prior on terminal-node parameters induces a normal prior on E(Y |x), calibrated so plausible regression values lie substantially within the observed response range.The calibration uses the observed minimum and maximum of Y, and BART is robust to exact hyperparameter choices when the prior covers the plausible range without excessive concentration or dispersion.
- Scale invariance: BART calibrates the response scale but does not require predictor standardization because tree splitting rules are invariant to monotone transformations of predictors.This contrasts with methods using linear combinations of predictors, such as neural nets, which require standardization choices for each predictor.
- Parameter prior calibration: Shrinking terminal-node parameters toward zero limits individual tree contributions, with tighter shrinkage as the number of trees or k increases.This prior shrinkage is described as the counterpart of the shrinkage parameter in Friedman’s (2001) gradient boosting algorithm.
3. Extracting information from the posterior.
Backfitting MCMC samples BART’s posterior by repeatedly updating each tree and the noise variance, with substantially better mixing than single-tree MCMC. After burn-in, posterior draws support point estimates, uncertainty intervals, partial-dependence effects, and model-free variable selection.
- Posterior sampling: Each Gibbs iteration updates every tree conditional on partial residuals from the other trees, then samples σ from its full conditional.The tree updates reduce to single-tree posterior draws using partial residuals, while σ is drawn from an inverse gamma distribution.
- Posterior sampling: Backfitting MCMC mixes dramatically better than single-tree MCMC, avoiding the latter’s tendency to become trapped near one large tree and supporting one long chain.Restarts of backfitting MCMC produced remarkably similar results even on difficult problems.
- Posterior inference: After burn-in, the ergodic chain yields an approximate dependent sample from p(f|y), enabling posterior means or medians for prediction and quantile-based posterior intervals.The number of iterations needed for reliable inference depends on the application and was relatively modest in the paper’s examples.
- Posterior inference: Posterior draws also estimate partial-dependence functions and their uncertainty intervals for selected predictors, including predictor combinations not necessarily observed in the data.The partial-dependence function averages fitted values over observed complements of the predictors of interest.
- Variable selection: BART performs model-free variable selection from predictor splitting frequencies, with smaller m favoring relevant variables and excluding predictors unrelated to y.When m is large, redundancy can mix irrelevant predictors into fitted trees; reducing m strengthens preference for relevant predictors.
4. BART probit for classification.
BART extends naturally to binary classification through a probit model in which probabilities are functions of the sum-of-regression-trees output. The extension retains regularized Bayesian backfitting with latent-variable augmentation for posterior inference.
- For binary Y, BART models p(x) = Φ[G(x)], obtaining classification probabilities from the sum of regression trees rather than majority or average voting.
- The classification extension requires only minor modifications: σ = 1, regularization on tree parameters, and Bayesian backfitting for posterior computation.
- Posterior computation introduces latent Zi ~ N(G(x),1), truncated by the observed class, and incorporates their simulation into the Bayesian backfitting algorithm.
- After burn-in, posterior draws of G and p(·) provide dependent samples for inference about the classification probability function.
5. Applications.
Applications show that BART performs strongly across real-data benchmarks and supports posterior inference, partial-dependence analysis, and variable selection. Simulations further indicate robustness to hyperparameter choices, reliable inference in high dimensions, and resistance to pure noise.
- 5.1 Real-data bake-off: BART-cv more often achieved smaller RMSE than competing methods across 42 real data sets, while BART-default was arguably the second-best overall.The comparison covered regression data sets with 3–28 numeric predictors and 0–6 categorical predictors.
- 5.2 Simulation studies: Posterior mean estimates and 90% intervals recovered the simulated regression function, with average frequentist coverage of 87% in-sample and 93% out-of-sample.Coverage declined for extreme x values because the prior exerted more shrinkage toward zero.
- 5.2 Simulation studies: BART’s partial-dependence estimates identified nonzero marginal effects for x1,...,x5 and zero effects for x6,...,x10, while posterior tree-use frequencies supported variable screening.These demonstrations used 5000 MCMC draws and showed how BART supports inference beyond estimating f(x).
- 5.2 Simulation studies: BART was robust to prior settings and tree counts: different settings gave similar results when k > 1, while too few trees performed poorly.Increasing the tree count beyond that needed for the fit caused only slight performance degradation.
- 5.2 Simulation studies: Out-of-sample fits were highly stable across settings and random seeds, with correlations of 0.9948 between contrasting settings and 0.9994 across replicate runs.This stability supports using one long MCMC run.
- 5.2 Simulation studies: With n = 100 and p = 20, 100, or 1000, BART retained good in-sample inference, became less reliable for out-of-sample extrapolation, and generally avoided pure-noise signals.For p = 10 and p = 100 under f ≡0, intervals covered or were close to zero; at p = 1000, the data became uninformative.
6. Execution time considerations.
BART’s execution time scales linearly with sample size for its short version but quadratically for the default version, while remaining nearly independent of the number of predictors. Compared with random forests, gradient boosting, and neural nets, BART has comparable scaling, with minimal BART fastest and default BART slowest.
- Default BART execution time grows quadratically with n, whereas the short version scales linearly because limiting iterations keeps trees small.For n = 100,000? No; the supplied experiment varies n from 100 to 10,000, and larger default-version datasets produce larger trees requiring more tree-based prediction operations.
- BART’s execution time is close to independent of p, especially relative to its much stronger dependence on n.This pattern was observed for the default version with p varied from 10 to 100 at each tested sample size.
- In practice, BART runtime may depend on signal complexity, which can require a longer burn-in period, and should account for tuning-parameter selection such as cross-validation.
- BART’s execution time is comparable to random forests, gradient boosting, and neural nets, with similar scaling across sample sizes; minimal BART is fastest and default BART slowest.These comparisons include training-set prediction generation for each method.
7. Extensions and related work.
BART’s sum-of-trees framework extends to models combining parametric terms, multivariate responses, and random effects while retaining modular MCMC fitting. The framework also supports model-free variable selection and motivates extensions such as SBART [Chipman, George and McCulloch (2007); Zhang, Shih and Muller (2007)].
- Extensions: BART extends to multivariate responses, yielding multivariate regression when predictors are shared and a generalization of Zellner’s SUR model [Zellner (1962)] when they differ.Each response-specific function is a sum of trees, with dependence represented through a p-dimensional covariance matrix.
- Extensions: BART can incorporate linear terms and random effects through an additional MCMC step that draws their associated parameters.The multivariate version is fit by drawing each response-specific tree-sum component.
- Extensions: BART’s variable-selection framework offers a model-free way to identify important features, while changing prior hyperparameters may further improve it.The default tree prior uses α = 0.95, assigning 5% prior probability to a single-node tree; smaller α values place more mass on small trees and may discourage modest-gain splits.
8. Discussion.
BART combines a sum-of-trees model, a regularization prior, and backfitting MCMC, with each tree contributing only part of the overall fit. Across diverse data sets and simulations, BART compared favorably with several competing methods, while its default version performed extremely well.
- BART’s essential components are a sum-of-trees model, a regularization prior, and a backfitting MCMC algorithm.
- Unlike single-tree Bayesian approaches, BART assigns each tree only part of the overall fit through a prior that shrinks tree effects toward simpler fits.
- BART compared favorably in out-of-sample predictive RMSE with boosting, the lasso, MARS, neural nets, and random forests across diverse data sets and simulations.
- The computationally inexpensive and easy-to-use default version of BART performed extremely well.