Source-linked AI summary
Large-scale benchmark study of survival prediction methods using multi-omics data
Moritz Herrmann, Philipp Probst, Roman Hornung, Vindi Jurinovic, Anne-Laure Boulesteix
TL;DR
Evidence was limited on whether multi-omics improves survival prediction and which methods suit these high-dimensional data. The study benchmarked methods across TCGA cancer datasets and found that only block forest outperformed the clinical-only Cox model on average for both performance metrics. Group-aware approaches generally performed better than naive approaches, though results varied across datasets.
Problem
It remained unclear whether multi-omics improves survival prediction and which methods are most appropriate for high-dimensional, grouped data.
Source: §1 · p. 1Method
The study benchmarked prediction methods on 18 TCGA cancer datasets using repeated cross-validation, comparing performance, group-structure strategies, and clinical-only references.
Source: §1 · p. 2, §3.1 · p. 4, §3.3 · p. 5Results
Only blockForest outperformed the clinical-only Cox model on average for both measures; mean cindex was 0.620 versus 0.618 and mean ibrier was 0.172 versus 0.175.
Source: §5 · p. 15Takeaways & Limitations
Group-aware learners performed better than naive learners in most dataset-metric comparisons, and prioritizing clinical variables may help retain established predictors.
Source: §4.4.3 · p. 13BlockForest's performance may be slightly optimistic because its variant was selected using a prior comparison that included 18 datasets reused here.
Source: §5 · p. 15
Abstract
from arXiv · showhide
Multi-omics data, that is, datasets containing different types of high-dimensional molecular variables (often in addition to classical clinical variables), are increasingly generated for the investigation of various diseases. Nevertheless, questions remain regarding the usefulness of multi-omics data for the prediction of disease outcomes such as survival time. It is also unclear which methods are most appropriate to derive such prediction models. We aim to give some answers to these questions by means of a large-scale benchmark study using real data. Different prediction methods from machine learning and statistics were applied on 18 multi-omics cancer datasets from the database "The Cancer Genome Atlas", containing from 35 to 1,000 observations and from 60,000 to 100,000 variables. The considered outcome was the (censored) survival time. Twelve methods based on boosting, penalized regression and random forest were compared, comprising both methods that do and that do not take the group structure of the omics variables into account. The Kaplan-Meier estimate and a Cox model using only clinical variables were used as reference methods. The methods were compared using several repetitions of 5-fold cross-validation. Uno's C-index and the integrated Brier-score served as performance metrics. The results show that, although multi-omics data can improve the prediction performance, this is not generally the case. Only the method block forest slightly outperformed the Cox model on average over all datasets. Taking into account the multi-omics structure improves the predictive performance and protects variables in low-dimensional groups - especially clinical variables - from not being included in the model. All analyses are reproducible using freely available R code.
1 Introduction
Multi-omics prediction combines high-dimensional molecular groups, often with clinical variables, but it remains unclear which methods best predict survival. This study benchmarks methods on real cancer data to assess predictive performance, group structure, and the added value of omics data.
1 Introduction
- Multi-omics datasets can exceed 10,000 or 100,000 variables, while their variables form groups that methods may account for.These settings require handling more variables than observations and may call for sparse, interpretable models.Source: §1 · pp. 1–2
- A large-scale systematic comparison using real multi-omics data was missing despite existing methods and smaller-scope comparisons.Earlier studies used limited numbers of methods or datasets, or simulated data.Source: §1 · p. 2
- 18 TCGA cancer datasets are used to benchmark survival-prediction methods and assess the added predictive value of multi-omics over clinical-only models.The study compares variants of penalized regression, statistical boosting, and random forest.Source: §1 · p. 2
2 Methods
The methods address survival prediction with high-dimensional, grouped multi-omics data, where performance is not the only practical consideration. The study reviews penalized regression, boosting, and random forests, including approaches that use group structure.
2.1 Preliminary remarks
- Group structure can encode prior knowledge about useful data types and help keep established clinical predictors from being lost among many omics variables.Sparse, interpretable models and transportability are also practical considerations; regression coefficients can ease interpretation and application to independent data.Source: §2.1 · p. 2
2.2 Penalized-regression-based methods
- Penalized regression regularizes maximum partial likelihood to address n << p, with Lasso producing sparse models but ignoring groups.TS IPF-Lasso allows group-specific penalties; priority-Lasso fits groups successively using earlier linear predictors as offsets, with this study deriving a surrogate priority order from separately fitted groups.Source: §2.2 · p. 3
2.3 Boosting-based methods
- Boosting iteratively updates a function by fitting weak base learners, with the number of steps and learning rate shaping updates and feature selection.Model-based boosting updates the loss-minimizing linear base learner, whereas likelihood-based boosting uses penalized likelihood and incorporates previous updates as offsets.Source: §2.3 · pp. 3–4
2.4 Random-forest-based methods
- Random forests average survival-tree predictions, while block forest modifies split-point selection to incorporate multi-omics group structure.For survival prediction, the forest averages cumulative hazard estimates across trees.Source: §2.4 · p. 4
3 Benchmark experiment
The benchmark compares prediction learners across TCGA datasets under a reproducible, repeated cross-validation design. It evaluates performance alongside sparsity and computation time, while distinguishing how methods use group structure.
3.1 Study design
- The study sought a neutral, reproducible, extendable comparison and selected datasets using prespecified inclusion criteria because method performance varies across datasets.The authors used the mlr framework to support a unified and extendable benchmark.Source: §3.1 · p. 4
3.2 Technicalities and implementation
- The benchmark compared 14 learners on 18 datasets, using repeated stratified 5-fold cross-validation and internal resampling for hyperparameter tuning.It used 10 repetitions for 11 smaller datasets and five repetitions for seven larger datasets; random forests used out-of-bag samples for inner resampling, while other learners used 10-fold cross-validation.Source: §3.2 · pp. 4–5
3.3 Performance evaluation
- Performance was assessed by integrated Brier score and Uno's C-index, model sparsity at variable and group levels, and training time.Random forests were not assessed for sparsity because they do not perform variable selection.Source: §3.3 · p. 5
3.4 Method configurations
- The learners span penalized regression, boosting, and random forests, with methods compared within three strategies for group information: none, group-aware without clinical priority, or group-aware with clinical priority.The clinical Cox model and Kaplan-Meier estimate served as reference methods.Source: §3.3 · p. 5, §3.4 · pp. 5, 7
3.5 Datasets
- The datasets contain five variable groups—clinical data and four molecular types—with about 80,000 to 103,000 molecular features per cancer type.After exclusions for incomplete data types and fewer than 5% effective cases, 18 datasets remained; clinical variables were selected using literature and availability criteria.Source: §3.5 · p. 7, Table 2 · p. 8
4.1 Failures and refinement of the study design
Model-fitting failures and software constraints affected the planned benchmark. In particular, SGL's computational burden restricted its results to four small datasets, and it was excluded from the rest of the analysis.
4.1 Failures and refinement of the study design
- When a learner failed in more than 20% of CV iterations for a dataset, failing iterations received random-prediction performance values (0.25 for ibrier; 0.5 for cindex); below 20%, successful-iteration performance means were used for all measures.For failures above 20%, computation time and number of selected features were assigned the means of the other iterations.Source: §4.1 · p. 8
- SGL failed under Windows 7 with the considered large feature counts, whereas it worked under Ubuntu.The rfsrc learner also required Windows 7 when run with batchtools parallelization.Source: §4.1 · p. 8
- SGL could not be run as planned because of extremely long fitting times, so results were available for only four small datasets and it was excluded from the remaining study.On those datasets, its mean cindex was 0.58 and mean ibrier was 0.24; models averaged 4,083 selected features and 5.7 hours per cross-validation iteration.Source: §4.1 · pp. 8–9, Table 3 · p. 9
4.2 Computation time
Training time varies by method and dataset size, with blockForest the slowest procedure. Sparse-event datasets COAD and KIRP also substantially increase computation time for IPF-Lasso and priority-Lasso.
4.2 Computation time
- BlockForest is the slowest method; standard Lasso and ranger are fastest, followed by glmboost and CoxBoost variants.IPF-Lasso, priority-Lasso, and GRridge take about 2 to 3 times as long as the fastest procedures.Source: §4.2 · p. 9
- Mean computation time is measured as model-fitting (training) time and averaged across outer-resampling iterations and datasets.Table 4 reports the average performance measures for each method.Source: §4.2 · p. 9
- Computation time depends on dataset size, and COAD and KIRP substantially increase IPF-Lasso and priority-Lasso runtimes despite having fewer cases.COAD has 17 events (9%) and KIRP 20 events (12%).Source: §4.2 · p. 9
4.3 Model sparsity
Model sparsity differs across methods at both the variable and feature-group levels. IPF-Lasso is especially sparse, while accounting for group structure generally retains variables from low-dimensional groups, including clinical variables.
4.3 Model sparsity
- Random forest variants are not assessed for variable-level sparsity because their models do not yield coefficients.Source: §4.3 · p. 9
4.3.1 Sparsity on the level of variables
- IPF-Lasso yields the sparsest models, averaging 7 variables; CoxBoost averages 10, while GRridge averages 984 and is not suited to variable selection in this setting.Lasso averages 16 variables, with higher variability; glmboost and priority-Lasso variants average more than 20.Source: §4.3.1 · p. 9
4.3.2 Sparsity on group level
- Methods accounting for group structure generally include variables from low-dimensional groups, whereas methods treating groups equally often omit clinical variables and select at most one miRNA variable.IPF-Lasso excludes cnv and rna features; CoxBoost favoring excludes miRNA features.Source: §4.3.2 · p. 11
- IPF-Lasso has strong group-level sparsity, while priority-Lasso cannot eliminate groups; GRridge selects variables from every group.Figure 2 shows selected-feature counts by group for learners with model coefficients and excludes GRridge because its models are much larger.Source: §4.3.2 · pp. 9, 11, Figure 2 · p. 10
4.4 Prediction performance
Across datasets, most learners beat Kaplan–Meier, but only blockForest beat the clinical Cox model; multi-omics gains were not general, while accounting for group structure tended to help.
4.4.2 Comparing prediction methods
- Naive learners showed metric-dependent rankings: random forest led on cindex and ranked among the best on ibrier, while all beat Kaplan–Meier.Likelihood-based boosting led on ibrier; model-based boosting performed worst on cindex.Source: §4.4.2 · p. 11
- Among group-structured methods not favoring clinical features, blockForest led on both measures; IPF-Lasso was sparsest and may suit settings prioritizing sparsity.Priority-Lasso led the penalized methods on cindex, while IPF-Lasso led them on ibrier.Source: §4.4.2 · p. 11
- When clinical features were favored, CoxBoost favoring outperformed priority-Lasso favoring and beat Kaplan–Meier on both measures; priority-Lasso favoring did so only on cindex.The comparison favors likelihood-based boosting over priority-Lasso in this setting.Source: §4.4.2 · p. 11
4.4.1 Overview and main findings
- Across datasets, most learners beat Kaplan–Meier, but only blockForest beat clinical Cox; group-aware methods generally surpassed naive methods on cindex.Figure 3 shows the distributions across datasets; the reported comparisons warrant caution because performance differences are small and variable.Source: §4.4.1 · p. 11, Figure 3 · p. 12, §4.4.2 · p. 13
4.4.2 Comparing prediction methods
- Small performance differences and substantial dataset variability mean claims that one method is superior should be treated cautiously.Paired tests found no significant differences for blockForest versus CoxBoost favoring or clinical-only Cox on either measure.Source: §4.4.2 · p. 13
4.4.3 Using multi-omics data
- Against clinical-only Cox, structured combined learners showed potential gains in some datasets, but six datasets had no omics learner outperforming Cox.The comparison excluded naive combined learners; the authors caution that differences were small in some cases.Source: §4.4.3 · p. 13
- Naive learners had better average performance than structured learners in only five cases; tests favored structured learners for cindex (p=0.0002), but not ibrier (p=0.0904).The five cases comprised four datasets for ibrier and only LGG for cindex.Source: §4.4.3 · p. 13
- Favoring clinical variables improved likelihood-based boosting on average cindex, moving it from among the worst to among the best methods; priority-Lasso's gain was smaller.The ibrier results did not confirm the improvement for priority-Lasso.Source: §4.4.3 · p. 13
5 Discussion
BlockForest had the strongest average predictive performance, but conclusions varied by measure and dataset, and practical method choice involves trade-offs beyond performance.
5 Discussion
- Table 5 summarizes added predictive value by dataset; method rankings vary with measure and dataset, and confidence intervals often overlap substantially.The authors caution that performance differences can be small; exemplary tests found no significant differences between BlockForest, CoxBoost favoring, and clinical-only Cox.Source: §5 · pp. 13, 15, Table 5 · p. 14
- BlockForest outperformed the other methods on average and was the only method to beat clinical Cox on both measures, but it was slow and not easily interpretable.CoxBoost favoring clinical features was faster, reasonably sparse, and only slightly worse on cindex or equal on ibrier.Source: §5 · p. 15
- Clinical-only Cox achieved mean cindex 0.618 and mean ibrier 0.175, versus BlockForest’s 0.620 and 0.172, while requiring a fraction of its computation time.The authors note that Cox may be preferable in practice because it is easily interpretable.Source: §5 · p. 15
- BlockForest is a strong choice when prediction performance is the main criterion and sparsity and interpretability are unimportant; other methods may suit different priorities.Priority-Lasso may perform better when accurate prior knowledge about variable groups is available.Source: §5 · p. 15
- Reusing datasets from the earlier BlockForest comparison could slightly favor BlockForest, although the authors consider noticeable bias unlikely.The studies differed in datasets, clinical variables, and feature selection for groups larger than 2,500 variables.Source: §5 · pp. 15–16
- Favoring clinical variables may be less useful or even detrimental when they contain less predictive information than in the benchmark datasets.The authors recommend considering this approach, but not applying it by default.Source: §5 · p. 16
- The benchmark could be extended with methods beyond those studied, more preprocessing approaches and datasets, and alternative handling of outer-resampling failures and missing values.Random forest was the only included method not relying on the proportional hazards assumption; the authors note that no widely used standardized approach addresses missing values in this context.Source: §5 · p. 16
Funding
The work was supported by the German Federal Ministry of Education and Research and the German Research Foundation.
Funding
- Funding came from the German Federal Ministry of Education and Research (BMBF) and the German Research Foundation.The listed grants were BMBF [01IS18036A] and German Research Foundation [BO3139/4-3 to A.-L.B., HO6422/1-2 to R.H.].Source: Funding · p. 16
Supplement: Method details
The supplement describes survival-model fitting through penalized regression, boosting, and random forests, with methods that incorporate multi-omics feature groups alongside methods that do not.
Supplement: Method details
- For survival outcomes, penalized-regression and boosting methods minimize negative partial log-likelihood rather than squared error, using individuals at risk at each event time and censoring indicators.Source: Supplement: Method details · p. 20
Penalized-regression-based methods
- Standard Lasso ignores group structure, while Sparse Group Lasso combines group- and feature-level penalties to induce sparsity at both levels.Source: Penalized-regression-based methods · pp. 20–21
- GRridge uses group-specific co-data, including multi-omics group structure, estimates group factors by empirical Bayes, and performs feature selection post hoc.Two-step IFP-Lasso derives group-specific penalty factors from a preliminary model; Priority-Lasso fits groups sequentially, using the previous group's linear predictor as an offset.Source: Penalized-regression-based methods · pp. 20–21
Boosting-based methods
- Model-based boosting updates the loss-minimizing feature-wise linear base learner, whereas likelihood-based boosting uses a penalized partial log-likelihood and applies shrinkage in coefficient estimation.Source: Boosting-based methods · pp. 21–22
Random-forest-based methods
- Random forests build trees from bootstrap samples using randomly selected features at each split, and the survival extension chooses splits by survival differences.Source: Random-forest-based methods · p. 22
2. Output the tree ensemble
The survival-forest procedure estimates cumulative hazard within terminal nodes and averages across trees for prediction. Block forest modifies split selection to account for multi-omics blocks.
2. Output the tree ensemble
- Random survival forest splits maximize survival differences, and terminal-node size is defined by a minimum number of deaths.The node-specific cumulative hazard is computed using the Nelson–Aalen estimator.Source: §2 · p. 22
- For prediction, the terminal-node cumulative hazard estimates are averaged across trees to form the ensemble cumulative hazard function.Source: §2 · p. 22
- Block forest incorporates multi-omics blocks by randomizing block choice and weighting split-criterion values blockwise during split-point selection.Source: §2 · p. 22