Practical considerations for specifying a super learner

• Choose a performance metric that matches the intended real-world task and reflects how closely predictions approximate the true function of interest.Source: Key messages · p. 2
• Use effective sample size as a rule of thumb for choosing the cross-validation scheme and shaping the SL algorithm library.Source: Key messages · p. 2
• Build a rich, diverse library that can adapt to varied functional forms while remaining computationally feasible and able to handle high-dimensional data.Source: Key messages · p. 2
• An ensemble SL combines predictions from multiple algorithms, whereas a discrete SL selects one best-performing algorithm by cross-validated performance.Source: Key messages · p. 2
• Justify a discrete SL as the final model by comparing individual library algorithms alongside one or more ensemble SLs.Source: Key messages · p. 2

Introduction

• Prediction functions support epidemiologic tasks including disease phenotyping, risk-factor evaluation, disease mapping, event forecasting, and causal-effect estimation.Source: Introduction · p. 2
• Machine-learning mortality scores have been shown to improve on conventional logistic-regression scores [1].Source: Introduction · p. 2
• Because the best learner cannot be known in advance, SL considers a user-specified set spanning parametric regressions and machine-learning algorithms.Source: Introduction · p. 2
• SL lets analysts include algorithms suggested by collaborators, related research, or subject-matter experts, and large-sample theory guarantees performance as well as possible among those specified algorithms [2].Source: Introduction · p. 2
• SuperLearner and sl3 provide a shared R interface to diverse algorithms, but analysts still need to make specification choices; the paper offers an overview and flowchart.Source: Introduction · p. 3

Overview of the super learner

• Stacking was proposed in 1992 [15], NNLS meta-learning was demonstrated in 1996 [16], and large-sample optimality theory later led to the name “Super Learner” [2].Source: Overview of the super learner · p. 3
• An ensemble SL can use a parametric or nonparametric meta-learning algorithm to combine candidate predictions; NNLS is one example, producing a weighted sum.Source: Overview of the super learner · p. 3
• Cross-validation estimates performance on novel data from the same distribution, and V-fold CV averages each candidate’s risk across validation sets.Source: Overview of the super learner · p. 3
• The article focuses on practical advice for tailoring the library, V-fold CV scheme, and discrete SL performance metric to data characteristics and substantive goals.Source: Overview of the super learner · p. 3
• The flowchart summarizes the process for specifying a super learner.Source: Figure 1 · p. 4

Defining a well-specified super learner

• The guidelines aim to tailor SL specification so it performs as well as possible for the prediction task at hand.Source: Defining a well-specified super learner · p. 5
• Recommendations depend on the information available in the data and the prediction problem.Source: Defining a well-specified super learner · p. 5
• A flowchart accompanies the heuristics as a concise guide to specifying the super learner.Source: Defining a well-specified super learner · p. 5

Preliminaries: Analytic dataset pre-processing

• The analytic dataset consists of observations on an outcome (Y) and covariates (X).SL requires this dataset to contain no missing values.Source: Preliminaries: Analytic dataset… · p. 5
• Consider omitting observations with extreme outcome values to avoid unduly influencing estimation of the prediction function.Source: Preliminaries: Analytic dataset… · p. 5
• Preprocessing may use outcome-blind covariate reductions, while reductions based on outcome–covariate relationships, such as lasso, belong within SL cross-validation.Examples of outcome-blind reductions include removing irrelevant, sparse, or constant covariates, or reducing highly correlated covariates.Source: Preliminaries: Analytic dataset… · p. 5

Flowchart step 1: Specify the performance metric for the discrete super learner

• Choose a performance metric that aligns with the intended real-world use of predictions and is optimized by the true prediction function.Source: Flowchart step 1: Specify the… · p. 5
• Metrics such as MSE, AUC, and NLL quantify the success of an estimated prediction function.Source: Flowchart step 1: Specify the… · p. 5
• Advanced users may define a custom risk function if its loss function is uniformly bounded [2].Truncating predicted values to the range of Y is given as a practical way to achieve boundedness.Source: Flowchart step 1: Specify the… · p. 5

Flowchart step 2: Derive the effective sample size

• Effective sample size is introduced as a proxy for information in the data, whose amount also depends on prediction-task complexity.Source: Flowchart step 2: Derive the effective… · p. 5
• For continuous outcomes, effective sample size equals the number of independent observations, or independent clusters when observations are clustered.Source: Flowchart step 2: Derive the effective… · p. 6
• For binary outcomes, information is limited by the minority class; effective sample size is the minimum of n and five times the minority-class count.Source: Flowchart step 2: Derive the effective… · p. 6

Flowchart step 3: Define the V-fold cross-validation scheme

• V-fold cross-validation partitions data into V validation sets and complementary training sets, evaluating each candidate on held-out observations.Its cross-validated risk estimates the true out-of-sample performance of a candidate trained on the full analytic dataset.Source: Flowchart step 3: Define the V-fold… · p. 6

Selecting the number of folds

• Larger V reduces bias but tends to increase variance in estimated cross-validation risk; theory requires V to grow more slowly than n, ruling out LOOCV for large n [16–18].The recommendation is to use LOOCV for very small effective sample sizes and larger V below n_eff as n_eff grows, while considering computational feasibility.Source: Selecting the number of folds · p. 6

Stratified cross-validation

• For binary outcomes, stratified V-fold cross-validation keeps outcome prevalence matched across training sets.Source: Stratified cross-validation · p. 6

Cross-validation with clustered data

• With clustered data, assign each cluster wholly to the same validation or training set and evaluate loss at the cluster level.Existing SL software supports this through its cluster-identifier “id” argument.Source: Cross-validation with clustered data · pp. 6–7

Flowchart step 4: Form library of candidate learners

• An ideal library combines diverse strategies, adapts to possible functional forms, remains computationally feasible, and handles high-dimensional data.Include parametric and highly data-adaptive learners when the effective sample size permits, tuning variants, and screeners for high-dimensional data.Source: Flowchart step 4: Form library of… · p. 7

Base learners

• Base learners specify a learning strategy, while tuning values turn them into fully specified candidates; diversity helps address unknown functional forms.Poor performers can receive zero weight in a dSL, while an eSL may combine their predictions with others.Source: Base learners · p. 7

Screener couplings

• When screening uses associations with the outcome, cross-validate it to avoid biasing estimated predictive performance.A screener can be paired with a candidate; for example, a lasso screener can pass selected covariates to a GLM within each training fold.Source: Screener couplings · p. 7

Smaller effective sample sizes

• With small effective sample sizes, structured learners such as parametric models or lower-depth trees are likely to perform better than highly adaptive learners.Neural networks beyond the simplest architectures generally require more data to fit their parameters well.Source: Smaller effective sample sizes · pp. 7–8

Risk functions

• A library may include algorithms that optimize a risk function different from the prediction task's metric, since NLL-minimizing algorithms can sometimes achieve higher AUC than AUC-maximizing ones.For a binary classification task evaluated by AUC, the dSL selects the candidate with the highest cross-validated AUC.Source: Risk functions · p. 8

Respecting the statistical model

• A diverse library can convey knowledge about the data-generating process and mitigate misspecification when that process is outside the statistical model.Known interactions can be represented explicitly with a regression formula or learned empirically by tree-based algorithms.Source: Respecting the statistical model · p. 8
• The dSL selects the candidate with the best cross-validated performance; an eSL can also be evaluated as a candidate alongside its component learners.If the eSL performs best, the dSL selects it; multiple eSLs with more flexible meta-learners can also be evaluated.Source: Respecting the statistical model · p. 8

Conclusion

• The article reviews choices for a well-specified super learner, offers data-adaptive recommendations, and presents a flowchart as a practical reference.Supplementary examples apply the flowchart and provide reproducible R code.Source: Conclusion · p. 9

Funding

• The work was funded by the US FDA, and its content does not necessarily represent or receive endorsement from the agency or government.Funding was pursuant to Contract 75F40119C10155.Source: Funding · p. 9

Conflicts of interest

• The glossary defines a candidate as a fully specified library algorithm, optionally paired with a screening algorithm.A screener returns a subset of covariates, which can be used to define a new candidate.Source: Conflicts of interest · p. 10
• Meta-level covariates are candidate-generated transformations of the input covariates, and the meta-learner uses them to predict the outcome.The meta-level dataset pairs cross-validated predictions with corresponding validation outcomes.Source: Conflicts of interest · p. 10
• A dSL selects the candidate with the best cross-validated performance, whereas an eSL combines predictions from multiple candidates.Figure 2 illustrates a three-fold CV specification with GLM, HAL, SVM, and GAM candidates and an NNLS meta-learner.Source: Conflicts of interest · p. 10, Figure 2 · p. 11