PaperlayerView exact revision on arXiv

Source-linked AI summary

Random forests for high-dimensional longitudinal data

Louis Capitaine, Robin Genuer, Rodolphe Thiébaut

arXiv:1901.11279v1stat.ME

TL;DR

High-dimensional longitudinal data combine many predictors with repeated observations, but random-forest approaches for these data were lacking. The paper extends mixed-effects random forests with a stochastic model and a forest that accounts for intra-individual covariance. Simulations favor the proposed approach, and the HIV-trial analysis selects transcripts associated with the interferon pathway; the study also notes limits tied to its random-effects choice and open theoretical questions.

  • Problem

    Random forests may need careful tuning when predictors greatly outnumber observations, and existing longitudinal approaches were limited despite repeated measurements and their correlation structure.

    Source: §1 · p. 2

  • Method

    The paper extends mixed-effects random forests with a stochastic model and proposes REEMforest, which aggregates REEMtrees fitted with intra-individual covariance.

    Source: §1 · p. 3, §2 · p. 3, §3.1 · p. 5

  • Results

    The simulations favored the proposed approach, and the HIV-trial analysis selected 21 transcripts, mainly associated with the interferon-α pathway.

    Source: §7 · p. 18, §6.3 · p. 18

  • Takeaways & Limitations

    The selected transcripts’ association with the interferon pathway is presented as supporting the relevance of the approach in this HIV-trial analysis.

    Source: §6.3 · p. 18, §7 · p. 18

    The vaccine-trial analysis used only a random intercept alongside the stochastic process, and theoretical analysis of these methods remains an open issue.

    Source: §7 · p. 18

Abstract

from arXiv · show

Random forests is a state-of-the-art supervised machine learning method which behaves well in high-dimensional settings although some limitations may happen when $p$, the number of predictors, is much larger than the number of observations $n$. Repeated measurements can help by offering additional information but no approach has been proposed for high-dimensional longitudinal data. Random forests have been adapted to standard (i.e., $n > p$) longitudinal data by using a semi-parametric mixed-effects model, in which the non-parametric part is estimated using random forests. We first propose a stochastic extension of the model which allows the covariance structure to vary over time. Furthermore, we develop a new method which takes intra-individual covariance into consideration to build the forest. Simulations reveal the superiority of our approach compared to existing ones. The method has been applied to an HIV vaccine trial including 17 HIV infected patients with 10 repeated measurements of 20000 gene transcripts and the blood concentration of human immunodeficiency virus RNA at the time of antiretroviral interruption. The approach selected 21 gene transcripts for which the association with HIV viral load was fully relevant and consistent with results observed during primary infection.

1 Introduction

Random forests can work in high-dimensional settings, but require careful tuning when predictors greatly outnumber observations. Repeated measurements offer a longitudinal setting with additional information, yet require methods that account for within-individual correlation; this paper extends mixed-effects random forests and evaluates them in simulations and an HIV vaccine trial.

1 Introduction

  • Random forests are non-parametric ensemble decision-tree methods used for prediction and classification, with good behavior in some settings where p exceeds n.The paper situates its work alongside theoretical results on consistency and asymptotic normality of random-forest predictions.Source: §1 · p. 2
  • When p is much larger than n, random forests may have limitations, and the number of variables sampled at each tree node must be carefully tuned.Prior approaches addressed high dimensionality through variable focusing or sparsity-inducing priors.Source: §1 · p. 2
  • Longitudinal random-forest methods were less developed, while repeated-measurement analyses require accounting for correlation through approaches such as mixed-effects models.Earlier longitudinal and clustered-data work used tree-based methods, and Hajjem et al. (2014) proposed a random-forest method using an EM algorithm.Source: §1 · p. 2
  • The paper extends mixed-effects methods with a stochastic model, develops a random-forest method for high-dimensional longitudinal data, and provides asymptotic prediction guarantees.It also compares methods in simulations and analyzes a therapeutic vaccine trial in HIV-infected patients.Source: §1 · pp. 2–3

2 The semi-parametric stochastic mixed effects model

The model represents each longitudinal response as a mean function of covariates plus individual random effects, a serial-correlation process, and measurement error. It generalizes earlier models by allowing a stochastic process and high-dimensional covariates.

2 The semi-parametric stochastic mixed effects model

  • The response for individual i at time t_ij is modeled as a covariate-dependent mean plus random effects, a stochastic process, and measurement error.The stochastic process captures serial correlation, while the random effects and process describe individual trajectories around the mean behavior.Source: §2 · p. 3
  • The model assumes independent Gaussian random effects, a centered Gaussian process with covariance depending on a parameter, and Gaussian measurement errors.The random-effects covariance is B, and the measurement-error variance is σ2.Source: §2 · p. 3
  • The model considers p covariates much larger than the total observation count and extends related stochastic models that used a one-dimensional time function or omitted the stochastic process.If the mean function is linear, the model reduces to the linear stochastic model of Diggle and Hutchinson (1989).Source: §2 · p. 3

3 Estimation

Estimation alternates between fitting the mean behavior function and updating variance parameters. The mean can be fit with several regression methods, including the proposed forest methods, while the variance updates use conditional expectations of maximum-likelihood estimators.

3 Estimation

  • The estimation procedure uses a variant of the EM algorithm, alternating between mean-function estimation and updates to variance parameters.The general algorithm initializes individual random-effect and process estimates and iterates these two steps.Source: §3 · p. 4

3.1 Mean behavior function estimation

  • At each mean-estimation step, variance parameters are held at their previous-iteration estimates, and the mean function can be fit using any regression method.The paper proposes SMERF as a stochastic generalization of MERF and REEMforest as an aggregation of REEMtrees fitted using a stochastic mixed-effects model.Source: §3.1 · pp. 4–5
  • REEMtree estimates tree-leaf values using intra-individual covariance, while REEMforest averages fitted REEMtrees to form its estimate.Each REEMtree leaf fit uses a stochastic linear mixed-effects model.Source: §3.1 · p. 5

3.2 Variance components estimation

  • Variance parameters are updated using conditional expectations of their maximum-likelihood estimators given the observed data.The updates include B, γ2, and σ2; the method also handles covariance functions with an additional parameter by evaluating a set of possible values.Source: §3.2 · pp. 5–7

4 Asymptotic analysis

The analysis establishes convergence of the fitted mean behavior under regularity conditions as the number of individuals grows. It also describes out-of-sample response prediction using fixed- and random-effects predictors and a conditional stochastic-process expectation.

4.1 Convergence of the expected outcome estimation

  • As the number of individuals grows, the fitted mean tends to the true mean behavior under regularity conditions on f.Source: §4.1 · p. 7

4.2 Convergence of out-of-sample outcome predictions

  • Out-of-sample response prediction uses fixed- and random-effects explanatory variables together with a conditional expectation of the stochastic process.Source: §4.2 · p. 7
  • Between observed times, the conditional process estimate interpolates between neighboring measurements; at or beyond the observed endpoints, its expectation is linear in Bias.Source: §4.2 · pp. 7–8

5 Simulation study

The simulations compare tree-based and forest methods in low- and high-dimensional longitudinal settings, including stochastic and non-stochastic outcome models. Forests generally had lower prediction errors than trees; REEMforest’s advantage over MERF was slight in the high-dimensional setting.

5.1.1 Explanatory variables

  • The simulated covariates included irrelevant variables and groups with time-varying behavior, with group-level shifts and additional time-dependent variability.The temporal patterns were motivated by transcriptomics data and examples from the DALIA trial.Source: §5.1.1 · p. 8

5.1.2 Outcome variable

  • The outcome simulations varied dimension by linking 2 variables to the response when p = 6 and 40 when p = 8000.They used non-stochastic and stochastic outcome models, with the latter adding ω_i to the response.Source: §5.1.2 · p. 9

5.2 Squared bias and prediction error

  • Prediction was assessed on held-out measurements, and squared biases were calculated for estimated quantities including f and model parameters.Test sets were formed by randomly selecting two measurements per individual.Source: §5.2 · pp. 9–10

5.3 Results

  • The simulations fixed n at 17 individuals and randomly assigned each individual 8–11 measurements, producing an unbalanced design.The low-dimensional comparison also included linear mixed-effects models; the high-dimensional comparison omitted LMEM as not adapted to that setting.Source: §5.3 · p. 10, §5.3.1 · p. 10, §5.4 · p. 13

5.3.1 A low-dimensional case

  • In the low-dimensional simulations, REEMforest’s squared bias for f was about 25% below MERF’s, and forests generally had lower biases than trees in stochastic-model comparisons.Using intra-individual covariance to evaluate leaf values reduced f bias more for forests than for CART; REEMforest also had slightly lower bias than MERF for other parameters.Source: Table 1 · p. 11, §5.3.1 · p. 12
  • In the low-dimensional simulations, MERF and REEMforest had the lowest prediction errors, with similar performance; this pattern also held for the stochastic models.Breiman’s RF performed worst, while LMEM had poor prediction ability because the mean function was not linear.Source: Figure 2 · p. 12, §5.3.1 · p. 12

5.4 A high-dimensional case

  • In the high-dimensional non-stochastic simulations, REEMforest’s squared biases were slightly lower than MERF’s, while the two forest methods had similar biases under the stochastic model.Forests systematically had lower biases than trees, especially for estimating f; REEMforest’s variable-importance scores were concentrated in the first three groups.Source: §5.4 · pp. 13–14, Table 2 · p. 13, Figure 4 · p. 14
  • In the high-dimensional simulations, forests had very low prediction errors compared with trees, while Breiman’s RF was competitive with trees, especially (S)MERT.The authors report that the gain from using RF rather than trees roughly compensated for Breiman’s RF not accounting for longitudinal structure.Source: Figure 3 · p. 14, §5.4 · p. 14

6 Application to the DALIA vaccine trial

The DALIA analysis used repeated measurements from 17 patients to relate gene transcripts to HIV viral-load dynamics after treatment interruption. SREEMforest improved prediction error over methods that did not model the longitudinal structure, and selected transcripts were mainly associated with the interferon pathway.

6 Application to the DALIA vaccine trial

  • SREEMforest and SMERF had lower prediction errors than simple RF and trees, with a slight advantage for SREEMforest.Breiman’s RF was comparable to MERF and REEMforest, which included only a random intercept.Source: §6 · p. 15, Figure 7 · p. 16

6.1 Variable selection using random forests

  • After SREEMforest converged, the authors subtracted estimated individual random effects and stochastic processes from the responses, then applied VSURF for variable selection.VSURF ranks variables by importance, thresholds low-importance variables, and selects nested models using out-of-bag error.Source: §6.1 · p. 16

6.2 Stability of the selected variables set

  • For neighborhood size 4, stability was around 0.5 except at mtry 1000; for neighborhoods larger than 8, it could exceed 75%.Stability was assessed among the 50 most important variables; rankings were described as quite consistent across a wide range of mtry values.Source: §6.2 · p. 17, Figure 8 · p. 17

6.3 Biological results

  • VSURF selected 21 variables after SREEMforest convergence, mainly transcripts associated with the interferon-α pathway.The transcripts were associated with interferon-annotated modules M1.2 and M3.4; EPSTI1 and SAMD9L were the two exceptions.Source: §6.3 · p. 18

7 Discussion

The paper presents a random-forest methodology for high-dimensional longitudinal data and reports favorable simulation results and an HIV-trial application. Its discussion highlights choices in mtry and random effects, alongside unresolved theoretical and methodological questions.

7 Discussion

  • The authors introduce a random-forest methodology for high-dimensional longitudinal data and report simulation superiority for their approach and its generalizations.They also apply the method to an HIV vaccine-trial dataset.Source: §7 · p. 18
  • The authors advise setting mtry roughly between 2p/3 and 3p/4, and not below p/2, because too-small values could prevent convergence and yield sub-optimal results.They link the recommendation to avoiding non-informative variable selection at tree nodes and stabilizing the inner RF in the iterative EM algorithm.Source: §7 · p. 18
  • The application used only a random intercept in addition to the stochastic process; with more individuals or fewer time points, the authors suggest considering time-point random effects.They expect this could increase model flexibility and capacity to estimate inter-individual variability.Source: §7 · p. 18
  • Theoretical analysis of non-parametric estimates embedded in an EM algorithm remains an open issue, and changing tree split criteria is proposed as another research direction.Source: §7 · p. 18

Appendix

The appendix presents the SREEMforest algorithm’s initialization and iterative updates, including fitting the mean behavior function and updating model parameters.

Appendix

  • The SREEMforest algorithm initializes its iteration counter and estimated random effects and model parameters.Source: Appendix · p. 19
  • At each iteration’s first step, the algorithm estimates the mean function from responses adjusted by prior random-effect and stochastic-process estimates, then fits and aggregates forest trees.Source: Appendix · p. 19
  • The algorithm’s second step updates the covariance and variance parameters using the current estimates of the mean function and individual effects.Source: Appendix · p. 19
Loading 1901.11279v1…