Rethinking How We Evaluate Methodological Progress in Health AI

1. Introduction

• The work addresses limited empirical evidence on whether comparisons from clinically meaningless tasks agree with those from expert-determined tasks, alongside concerns about reproducibility and task definition.Prior work had documented reproduction-related reversals in reported neural-network gains over gradient-boosted trees.Source: §1 · pp. 1–2
• The strongest overall model is a gradient-boosted tree paired with an extremely wide, sparse tabularization of the full feature space.The authors also report preliminary evidence of shared structure between tasks, datasets, and algorithms, particularly on clinically meaningful tasks.Source: §1 · p. 2
• A shared MEDS interface supports reproducible, standard-interface reimplementations of twelve algorithms for baseline comparisons across MEDS datasets.The implementations are intended to let future methodological research extend rather than recreate these comparisons.Source: §1 · p. 2
• The study compares algorithm performance in discriminative and computational terms across tasks and two datasets, offering guidance on likely performance and computational cost.Source: §1 · p. 2
• Randomly generated tasks provide highly reliable estimates of relative algorithm performance, including across datasets.This offers a more accessible route to robust methodological research in EHR AI.Source: §1 · p. 2
• Relating task properties to algorithm and dataset properties yields preliminary evidence that algorithm rankings are partly predictable from evaluation-setting characteristics.The authors identify this as a source of further methodological insight.Source: §1 · p. 2

2. Methods

• Algorithms are trained for binary classification through a consistent interface, then evaluated by held-out-subject AUROC across two task families and two public critical-care EHR datasets.The analysis compares rankings across settings, relates ranking variance to task, dataset, and algorithm properties, and examines performance against models’ first public artifact dates.Source: §2 · pp. 2–3

2.1. Models

• Twelve models were reimplemented with a fixed interface and staged verification, while five model families were selected from 49 approaches identified through literature review.The selected set also included MEDS-Tab configurations, an ICU-XGBoost baseline, and recent competitors MOTOR and MEDS-EIC-AR.Source: §2.1 · p. 3

2.2. Tasks

• Each dataset contributes 10 clinically meaningful and 10 randomly generated binary prediction tasks; generated tasks pair event codes with prediction horizons, while clinically meaningful tasks specify triggers, prediction-time anchors, horizons, and sometimes exclusions.Generated tasks require min(npos, nneg) ≥150; clinically meaningful tasks use a reporting floor of min(npos, nneg) ≥40.Source: §2.2 · p. 4

3.1. Q1: Comparative conclusions generally transfer across settings

• Randomly generated tasks can serve as an efficient first-pass substrate for broad modeling comparisons, but clinically meaningful tasks remain essential for task-specific or deployment-oriented evaluation.Source: §3.1 · p. 4
• Ranking agreement across settings ranged from 76% to 92%, strongest for MIMIC-IV and NWICU randomly generated tasks (92%) and MIMIC-IV generated versus clinical tasks (91%).Agreement was statistically significantly higher than chance in all six setting comparisons after Holm correction.Source: §3.1 · p. 4, §3.2 · p. 6

3.2. Q2: Dataset-task-algorithm interactions provide preliminary evidence of shared structure

• Clinical-task pairwise comparisons reversed aggregate rankings more often than generated-task comparisons: 15.2% versus 8.3% on MIMIC-IV and 16.7% versus 10.9% on NWICU.Source: §3.2 · pp. 5–6
• Interaction terms accounted for roughly 24% of AUROC variance on clinically meaningful tasks, compared with below 10% on randomly generated tasks.Source: §3.2 · p. 6
• Across six clinical task concepts present in both datasets, same-concept rankings were more similar than different-concept rankings (mean Kendall’s τb 0.601 versus 0.437; exact permutation p = 0.019).This provides preliminary evidence that task conceptual identity is associated with performance differences that persist across datasets.Source: §3.2 · pp. 5–6

3.3. Q3: Methodological progress can arise from new uses of existing techniques, not only new architectures

• MEDS-Tab-large, an XGBoost-derived algorithm, achieved best-in-class performance with 5,983,579 MIMIC-IV and 1,103,767 NWICU features from wide, sparse tabularization.The controlled comparison found mixed progress over the decade, poorly summarized by architecture recency.Source: §3.3 · pp. 6–7
• MEDS-Tab-large occupied the high-cost end of the runtime–performance frontier, while neural MEDS-EIC-AR offered competitive performance at greatly reduced wall-time cost.Source: §3.3 · p. 7

4.1. Implications for methodological research in Health AI

• A two-stage evaluation strategy first measures algorithm performance that transfers across task families and datasets, then studies systematic departures where task-, dataset-, and design-specific knowledge may reside.The first stage can be measured cheaply; the second examines the variation that may explain where algorithms work best.Source: §4.1 · p. 7
• New algorithms should be assessed not only by benchmark improvement, but also by whether gains transfer, where they fail, and what those failures reveal about which algorithms work where.The discussion proposes characterizing task subsets and testing whether properties such as horizon, prevalence, clinical category, or cohort size predict their rankings.Source: §4.1 · p. 7
• The authors report that recent generative AI and shared tooling made reconstructing heterogeneous pipelines under a common interface feasible with comparable effort per model.They distinguish standardized re-implementation from exact reproduction and intend to contribute the implementations to MEDS-DEV.Source: §4.1 · pp. 7–8

4.2. Limitations and future work

• Conclusions are limited to two structured hospital and ICU EHR datasets, a subset of algorithms, ten tasks per setting, and best-effort re-implementations.The uncertainty analysis also omits repeated training across random seeds and focuses primarily on AUROC.Source: §4.2 · p. 8

5. Related work

• Prior work groups barriers to cumulative Health AI progress around clinical data, meaningful and comparable tasks, and reproducibility and comparability of results [1] [17].Shared datasets alone do not guarantee comparable experiments, and benchmark citation does not ensure sufficiently similar setups.Source: §5 · p. 8
• EHRSHOT [33] and FoMoH [23] expand standardized evaluation across longitudinal or diverse clinically meaningful tasks, while this work examines how conclusions depend on dataset, cohort, and task choices.The authors describe their work as complementary to fixed benchmarks rather than another fixed benchmark.Source: §5 · p. 8

Appendix A. List of considered models

• The considered-model list includes ICU/EHR models such as BEHRT, Med-BERT, STraTS, MOTOR, and PULSE-ICU, alongside other approaches including EveryQuery and Delphi-2M.The lists provide publication or code-resource information where available.Source: Appendix A · pp. 12–13
• The appendix surveys 49 structured-EHR model cards grouped by training strategy: supervised, pretrained, and other zero-shot or prior-fitted approaches.For zero-shot models, the table distinguishes direct prediction from prediction through sampled or materialized future trajectories.Source: Table 2 · p. 12

B.1. Statistical analysis overview

• The primary comparison is the held-out AUROC difference between two algorithms on a task and evaluation setting; aggregate rankings are used to assess how conclusions change across settings.Ranking agreement is the share of algorithm pairs with the same mean-AUROC difference sign in both settings, tested against chance with permutation tests and Holm adjustment.Source: §B.1 · p. 14
• The analysis also measures task–algorithm interactions and relates aggregate performance to model-family artifact dates.Within-setting task consistency is one minus the task-level reversal rate; the appendix also describes a decomposition into algorithm, task, and interaction components.Source: §B.1 · p. 14

B.2. AUROC and pairwise algorithm differences

• AUROC is interpreted as the probability that a randomly selected positive example receives a higher score than a randomly selected negative example.Positive task-level AUROC differences indicate that algorithm i outperforms algorithm j on that task.Source: §B.2 · p. 14
• For setting-level comparisons, the authors average AUROC across the setting’s tasks and define aggregate pairwise differences from those averages.These aggregate comparisons provide the reference for task-level reversals.Source: §B.2 · p. 14

B.3. Task-level reversal rate

• A task-level reversal occurs when a task-specific algorithm-pair comparison points in the opposite direction from the aggregate comparison for that setting.The reversal rate is the proportion of task–algorithm-pair comparisons that disagree; ties are excluded.Source: §B.3 · p. 14

B.4. Uncertainty quantification

• Uncertainty intervals combine patient-bootstrap draws with task resampling, reusing a common patient draw across tasks and algorithms within each dataset.Marginal 95% percentile intervals assume exchangeable tasks within settings and condition on fitted algorithms; repeated training was not bootstrapped because of computational cost.Source: §B.4 · pp. 14–15

Appendix C. Dataset characterization

• Dataset characterization is presented in Table 3.Source: Appendix C · p. 15

D.2. Task adequacy threshold.

• The minority-class count limits pairwise discrimination; N = 150 corresponds to an approximate AUROC standard-error scale of 0.041, versus 0.079 at N = 40.These values guide task selection rather than serving as exact confidence intervals; actual AUROC sampling variance depends on both class sizes and model-score distributions.Source: §D.2 · p. 15

D.3. Matched tasks

• Conceptually matched clinical task pairs for the cross-dataset ranking-similarity analysis are summarized in Table 8.Source: §D.3 · p. 15

Appendix E. Model development

• Model implementations used Claude Code v2.1 with Claude Opus 5 agents, without raw PhysioNet data access, on a cluster with NVIDIA L40S GPUs.Source: Appendix E · p. 15

Appendix F. Detailed results

• Detailed AUROC results are reported in Tables 9–12, and compute costs in Tables 13–15.Source: Appendix F · p. 15

Appendix G. Model recency

• Appendix G begins a section on model recency.Source: Appendix F · p. 15

• The generated-task tables report held-out statistics, and task selection requires min(npos, nneg) ≥150; average AUROC is across 12 models.Source: methodological progress · p. 16, Table 4 · p. 16, Table 5 · p. 16
• Some performance improvement appears over time by first public artifact date, but several recent models trail older approaches and MEDS-Tab-Large is strongest overall.Source: methodological progress · pp. 15–16
• Strong performance is already achieved by older components such as logistic regression and XGBoost; newer components do not consistently improve on that frontier.The component-date curves are retrospective summaries of the evaluated models, and component dates are editorial choices.Source: methodological progress · pp. 15–16
• MEDS-Tab-Large pairs XGBoost with a modern, extremely wide and sparse tabularization of the longitudinal record rather than a recently introduced learning algorithm.Its performance highlights that newer systems may benefit from how clinical data are represented and exposed to a model.Source: methodological progress · p. 16

• The comparison uses generated-task macro-AUROC and amortized wall time, including task-specific preprocessing, training, and required feature extraction.Shared preprocessing and pretraining costs are amortized across tasks; the Pareto frontier marks algorithms offering higher performance at a given cost or lower cost at a given performance.Source: Appendix H · pp. 16–17
• MEDS-Tab-Large achieves the highest mean AUROC at the second-highest cost, while MEDS-EIC-AR and RETAIN occupy lower-cost points on the frontier.The frontier includes algorithms from several generations and model families, rather than forming a monotonic sequence of increasingly recent architectures.Source: Appendix H · p. 17
• Inexpensive historical approaches such as ICU-XGBoost and RETAIN achieve relatively strong performance at very low cost, while higher-performing algorithms generally require more computation.Source: Appendix H · p. 17

Additional analyses

• Table 17 reports aggregate pairwise agreement and Kendall rank correlation between MIMIC-IV Generated and other evaluation settings.The appendix describes these as comparisons with MIMIC-IV Generated; the supplied caption identifies the table's scope.Source: Appendix I · p. 17, Table 17 · p. 21
• Tables 9–12 report task-level AUROC for generated and clinical tasks in MIMIC-IV and NWICU, with unweighted task means and point estimates.Bold marks the highest unrounded value in each column.Source: Table 9 · p. 18, Table 10 · p. 18, Table 11 · p. 19
• Tables 18 and 19 summarize task–algorithm interaction through individual-task reversals and a decomposition of AUROC variation.The reversal analysis covers 660 task–model-pair comparisons per setting.Source: Appendix I · p. 18, Table 18 · p. 22, Table 19
• Table 20 examines whether matched clinical-task concepts produce more similar cross-dataset model rankings than incorrect correspondences.The comparison uses the same six task concepts and also reports a broader nonmatched comparison.Source: Appendix I · p. 18, Table 20 · p. 22
• The horizon analyses assess relative MOTOR-FT versus MEDS-EIC-AR performance and BEHRT absolute AUROC across generated tasks.Table 22 uses Spearman correlation across 10 tasks per dataset; positive ρ denotes a growing relative MOTOR-FT advantage at longer horizons.Source: Appendix I · pp. 18, 22, Table 21 · p. 22, Table 22 · p. 23
• Table 23 compares fine-tuning with linear probing for MOTOR and pretrained with supervised-only MEDS-EIC-AR across settings and overall.MOTOR differences are FT minus LP; EIC-AR differences are pretrained minus supervised-only.Source: Appendix I · pp. 18, 23, Table 23 · p. 23
• The correspondence-permutation test across six task concepts yields p = 0.0194.The test considers 720 bijections.Source: Appendix I · p. 22
• The compute analysis reports macro-AUROC and pipeline wall time for retained generated tasks, with pooled means equally weighting tasks across datasets.It includes 8 MIMIC-IV and 7 NWICU tasks; shared preprocessing and pretraining are amortized, and times are not hardware-normalized.Source: Table 13 · p. 20, Table 14 · p. 20, Table 15 · p. 21