Bayesian additive regression trees and the General BART model

1 Introduction

• BART’s widespread use as a comparison model and growing interest in flexible or robust methods motivate an in-depth tutorial.The authors aim to help researchers understand the method and diagnose likely problems when unexpected results occur.Source: §1 · p. 3
• General BART unifies extensions that subtract a latent variable from the outcome and model the residual with BART.The authors describe this shared construction as allowing extensions without rewriting or re-deriving the MCMC procedure for drawing regression trees.Source: §1 · pp. 3–4
• The tutorial reviews what gives BART flexibility, why it is a sum of regression trees, and the mechanics of its algorithm.The paper presents this review as addressing commonly asked questions about BART.Source: §1 · p. 4
• BART outperforms Bayesian linear regression when data are generated from a complicated model; the paper also applies it to continuous- and binary-outcome datasets.The introduction identifies one real-life dataset with continuous outcomes and another with binary outcomes.Source: §1 · p. 4
• General BART extends BART to semiparametric models, correlated outcomes, survey settings, and cases needing weaker error assumptions.The introduction describes how priors and posterior distributions are specified for these extension examples.Source: §1 · p. 4

2.1.1 Formal definition

• BART targets prediction of complex relationships between covariates and a continuous outcome using Y = f(X) + ε_i, with f specified as a sum of regression trees.Each tree has a binary structure and associated terminal-node parameters.Source: §2.1.1 · p. 5

2.1.2 Single regression tree

• A single regression tree routes covariate values through split conditions to a terminal node, whose parameter supplies the subject’s predicted outcome.Viewed as an ANOVA model, a tree can represent both main effects and multi-way interactions.Source: §2.1.2 · pp. 6, 8

2.1.3 Sum of regression trees

• Summing stepwise regression trees lets BART approximate nonlinear main effects and multi-way interactions without requiring researchers to specify them.Each tree contributes a stepwise function; their sum can form a more complex function.Source: §2.1.3 · pp. 10–11
• Because the mean function is unknown, BART uses priors for the tree functions, allowing posterior predictions to account for uncertainty in functional form and parameters.The tree structures, split variables and values, and terminal-node means are unknown in practice.Source: §2.1.3 · p. 11

2.1.4 BART machinery: a visual perspective

• A visual walkthrough follows four regression trees through five MCMC iterations, beginning with each tree initialized as a single root node.The figures illustrate the tree structures as the MCMC steps proceed.Source: §2.1.4 · p. 11, Figure 3 · p. 12, Figure 4 · p. 14
• For each tree, BART fits the residual after excluding the other trees, proposes a new structure with Metropolis–Hastings, and accepts it using likelihood and tree-move probabilities.After the structure update, BART draws that tree’s terminal-node parameters and proceeds to the next tree.Source: §2.1.4 · p. 15
• Each full MCMC draw predicts Y by summing terminal-node means; posterior draws also yield 95% prediction intervals, after a burn-in typically lasting 100 to 1000 iterations.The illustrative trees remain shallow, with a maximum depth of 3.Source: §2.1.4 · p. 16

2.1.5 A rigorous perspective on the BART algorithm

• The formal specification assumes tree-and-leaf pairs are mutually independent and independent of σ, with independent terminal-node parameters and Normal and inverse-gamma priors.The passage describes terminal-node means as Normal and σ as inverse gamma.Source: §2.1.5 · p. 17
• The tree prior sets split probability to α(1+d)^β; larger α makes splitting more likely, while larger β reduces terminal-node counts, and uniform covariate selection does not promote variable selection.Uniform selection is also the suggested default for choosing a split cutoff.Source: §2.1.5 · pp. 17–18
• The depth-penalizing tree prior discourages overfitting and supports convergence to target function f(X) (Roˇckov´a and Saha, 2018), while combining shallow trees into a stronger model.The prior limits the growth of individual trees so that many shallow trees can be summed together.Source: §2.1.5 · pp. 17–18
• BART proposes each new tree through four local moves: grow, prune, swap, or change a nonterminal node’s splitting criterion.Grow splits a terminal node; prune combines terminal siblings; swap exchanges the splitting criteria of two nonterminal nodes.Source: §2.1.5 · p. 19
• After drawing a tree conditional on its residuals and σ, BART draws terminal-node parameters conditional on the tree and residuals.Each terminal node is associated with the residuals allocated to it and their count.Source: §2.1.5 · pp. 19–20

2.2 Binary outcomes

• For binary outcomes, BART uses a probit link and Albert and Chib (1993) data augmentation: latent Z values are drawn from truncated normals, modeled as continuous BART with variance fixed at 1, and updated iteratively until convergence.Source: §2.2 · pp. 20–21

3.1 Posterior performance via synthetic data

• At m=50, BART improved bias over BLR and m=1, narrowed 95% intervals, and posterior estimates and intervals stabilized as m increased.The comparison is shown in Figure 5; the authors note little difference between m=200 and m=20,000.Source: Figure 5 · p. 22, §3.1 · p. 23
• BLR point estimates were often far from the true values and many true values fell outside its 95% intervals; m=1 BART mostly covered them but remained biased.Source: §3.1 · p. 23

cility Compare dataset

• The dialysis-facility example predicted log SHR from facility characteristics and quality measures, comparing MLR, RF, and BART with 10-fold RMSE.The dataset included 105 quality measures and 21 facility characteristics.Source: cility Compare dataset · pp. 23, 25
• BART and RF had similar log-SHR prediction performance and both outperformed MLR; their mean RMSE was 0.23 versus 0.24 for MLR.Figure 6 shows the 10-fold cross-validation RMSEs.Source: cility Compare dataset · p. 25, Figure 6 · p. 26

3.3 Predicting left turn stops at an intersection

• The 6-meter windows were selected as best-performing; the first three PCs explained nearly 99% of variance, and binary predictions were evaluated by 10-fold AUC.Source: §3.3 · pp. 27–28
• At 50 meters from the intersection center, the dataset comprised the first three principal components of vehicle speed from 50 to 56 meters, with future stopping as the outcome.A stop was defined as vehicle speed below 1 m/s.Source: §3.3 · p. 28
• Across the methods evaluated, BART achieved the highest mean AUC in 10-fold cross-validation, 0.81, compared with 0.79 for logistic regression and 0.78 for RF.Source: §3.3 · pp. 29–30

4 General BART model

• The framework combines BART trees on X with H(W,Θ) and flexible errors G(Σ), while Gibbs updates reuse standard BART tree draws on Y−H(W,Θ).Tree updates use an outcome modified by subtracting H(W,Θ); draws for Θ and Σ depend on the chosen H function and error model.Source: §4 · pp. 30–31

4.1 Semiparametric BART

• Zeldow et al. (2018)'s semiparametric BART models effects of interest linearly, with at most simple interactions, while flexibly modeling nuisance or confounder variables.With a multivariate normal prior on coefficients, standard Bayesian linear regression can produce their posterior draws.Source: §4.1 · p. 33

4.2 Random intercept BART for correlated outcomes

• Tan et al. (2018b)'s random intercept BART handles correlated continuous or binary outcomes, with correlated binary outcomes as its main focus.It adds subject-specific random intercepts with a shared variance component to the BART model.Source: §4.2 · pp. 33–34

4.3 Spatial BART for a statistical matched problem

• Zhang et al. (2007)'s Spatial BART addresses survey statistical matching, modeling relationships between variables collected in different datasets for overlapping subjects with a conditional autoregressive prior.The prior uses an adjacency matrix to represent neighboring groups.Source: §4.3 · pp. 34–35

4.4 Dirichlet Process Mixture BART

• George et al. (2018)'s Dirichlet Process Mixture BART relaxes the error-distribution assumption by mixing subject-specific error means and variances, with fewer than n unique components for identifiability.The model does not assume that the subject-specific means and errors are independent.Source: §4.4 · pp. 35–36
• DPMBART sets ν=10, uses a 0.95 quantile in choosing λ, and sets µ0=0 because it subtracts the mean of Y.The k0 choice uses multiple-linear-regression residuals to locate µ, with ks=10.Source: §4.4 · pp. 36–37
• The prior for α follows Rossi (2014) and relates α to the number of unique components in (ai, σ²).The final α draw conditions on (ai, σ²), placing α on a grid and applying Bayes’ theorem.Source: §4.4 · p. 37
• For DPMBART, the posterior draw first draws (T, M)|a_i, σ² using the idea of Equation (10), viewed as a weighted BART draw with ε ∼ N(0, w_iσ²), then draws (a_i, σ²)|(T, M) following Equation (11).Source: §4.4 · p. 37

5 Discussion

• The tutorial walks through BART’s model and algorithm and clarifies its components for research applications.Source: §5 · p. 37
• BART’s uniform prior for choosing split covariates prevents variable selection; alternatives include different covariate priors, a Dirichlet process prior, and a spike-and-slab prior.The cited alternatives are Kapelner and Bleich (2016), Linero (2018), and Liu et al. (2018), respectively.Source: §5 · pp. 37–38
• BART can be slow because it requires many MH steps, especially with large n or p; proposed responses include parallelizing computation or improving MH-step efficiency.Source: §5 · p. 38
• Other reported limitations include underestimated uncertainty from inefficient mixing when true variation is small, difficulty handling smooth functions, and spurious interactions with many covariates.Source: §5 · p. 38
• General BART offers a simple way to extend BART without rewriting the MCMC algorithm that draws regression trees.The framework is described as flexible enough for many extensions of interest to researchers.Source: §5 · pp. 38–39
• The framework unifies extensions for semiparametric models, correlated outcomes, survey statistical matching, and weaker error-distribution assumptions.The tutorial presents these extensions as related within a single framework.Source: §5 · p. 39
• Examples include modeling longitudinal random effects with BART and combining correlated outcomes, an interpretable linear component, and robust error assumptions.Source: §5 · p. 39
• General BART depends on rewriting the model so tree draws can be separated from the rest; mapping terminal nodes to regression models may require rewriting that procedure.Source: §5 · p. 39

A Hyperparameters for BART

• For continuous outcomes, hyperparameters mix fixed and data-driven choices, with v chosen so the prior for the sum of tree parameters covers the outcome range.The appendix also describes rescaling outcomes to (−0.5, 0.5) to facilitate posterior calculations.Source: A Hyperparameters for BART · pp. 46–47
• For both outcome types, the default split-prior values α = 0.95 and β = 2 provide a balanced penalizing effect on node-splitting probability.Source: A Hyperparameters for BART · pp. 46–47
• For binary outcomes, the μμ and σμ hyperparameters differ from the continuous-outcome settings, and the latent variable is transformed.Source: A Hyperparameters for BART · p. 47

B.1 P(µkj|Tj, σ, Rj)

• R_kj is the subset of R_j allocated to terminal node k, containing the residuals for subjects assigned to that node.Source: §B.1 · p. 47
• Within a terminal node, the allocated residuals are modeled as normal with mean μ_kj and variance σ², with a prior for μ_kj conditional on the tree.Source: §B.1 · p. 47
• The section then gives the posterior distribution for μ_kj and identifies the residual sum of squared differences used in its calculation.Source: §B.1 · p. 48

B.2 P(σ2|(T1, M1), . . . , (Tm, Mm), Y )

• The variance parameter σ² is assigned an inverse-gamma prior.Source: §B.2 · p. 48
• Its posterior is proportional to the likelihood of Y given the trees and σ, multiplied by the prior on σ².Source: §B.2 · p. 48
• The BART prediction assigned to an observed outcome is the sum of the predictions from the trees.Source: §B.2 · p. 48

• The section is modified from Appendix A of Kapelner and Bleich (2016).Source: step · p. 49
• The grow/prune Metropolis–Hastings ratio is organized into transition, likelihood, and tree-structure ratios.Source: step · p. 49
• The section gives explicit formulas for each ratio under the grow and prune proposals.Source: step · p. 49

C.1.1 Transition ratio

• The grow proposal uses a researcher-chosen P(grow), defaulting to 0.25, and accounts for available terminal nodes, variables, and split values.Source: §C.1.1 · p. 49
• The transition ratio accounts for the probability of the move and selecting the internal node to prune, among nodes with two terminal children.Source: §C.1.1 · pp. 49–50

C.1.2 Likelihood ratio

• For the likelihood ratio, only the terminal node where the two children are grown needs to be considered because the rest of the tree is unchanged.Source: §C.1.2 · p. 50

C.1.3 Tree structure ratio

• The tree-structure ratio uses node-splitting probabilities proportional to α(1+dθ)^β and probabilities for selecting the split variable and value.Source: §C.1.3 · pp. 50–51

C.2 Prune proposal

• The prune proposal’s Metropolis–Hastings ratio is the inverse of the grow proposal’s ratio.Source: §C.2 · p. 51