PaperlayerView exact revision on arXiv

Source-linked AI summary

Alert-List Concordance for Evaluating Imputation Strategies in ICU Sepsis Prediction

Priyanka Siddappa, Muntaqim Ahmed Raju, Shifat Haider Al Amin, Ruizhe Ma

arXiv:2610.22635v1cs.LGq-bio.QM

TL;DR

Aggregate metrics do not show whether imputation pipelines flag the same patients, even though clinical alerts act on patient lists. The paper compares top-k alert-set overlap across imputation strategies and finds that similar AUROC can accompany divergent alerts, with ForwardFill a separate outlier.

  • Problem

    Reconstruction error and aggregate discrimination do not directly measure whether pipelines flag the same patients, although alerts are issued to a limited share of the census.

    Source: §I · p. 1

  • Method

    The paper introduces alert-list concordance, comparing Jaccard overlap of top-k alert sets across imputation pipelines, with patient-level bootstrap intervals.

    Source: §I · p. 1

  • Results

    Among 22 reconstruction-method pairs without statistically significant patient-level AUROC differences after Bonferroni correction, top-decile Jaccard ranged from 0.518 to 0.750; ForwardFill had Jaccard 0.40 to 0.45 against other methods.

    Source: §VIII · p. 8

  • Takeaways & Limitations

    Pipeline changes can move patients on and off the alert list without visible AUROC change, so overlap with the prior deployment should be evaluated.

    Source: §VI · p. 7

    The study shows that imputation changes who is flagged, not that this changes escalation decisions or outcomes.

    Source: §VII · p. 7

Abstract

from arXiv · show

Preprocessing is an under-measured reliability risk in clinical prediction: a model can keep a similar AUROC and still change who receives an alert. Imputation quality is commonly evaluated using reconstruction error or aggregate discrimination, neither of which directly measures agreement on which patients are flagged. At a matched alert rate, pipelines with similar aggregate performance can produce different patient lists. This question is especially relevant under the frequent, structured missingness of real ICU records. We therefore introduce alert-list concordance, which asks whether two pipelines identify the same high-risk patients, measured by the Jaccard similarity of their top-k% alert sets with patient-level bootstrap confidence intervals. On PhysioNet/CinC 2019 (40,336 patients, 71.9% missingness), we compare nine imputation strategies and a native-missingness baseline using a fixed XGBoost configuration, fitted and Platt-calibrated separately for each strategy. After Bonferroni correction, 22 of the 28 reconstruction-method pairs showed no statistically significant difference in patient-level AUROC, yet their alert lists overlap at Jaccard 0.518 to 0.750. Similar aggregate discrimination can therefore accompany different patient selections. ForwardFill is a separate outlier, pairing higher discrimination with the lowest overlap in the matrix, 0.40 to 0.45. The broad pattern holds within each hospital system and under six patient-level aggregation rules. At the row level, eight of ten methods fall within an AUROC range of 0.011. In this row-level pipeline, deep imputers did not outperform mean imputation on discrimination, and the sequential implementations cost 5 to 12 times mean imputation in total pipeline wall-clock time on our hardware.

I. INTRODUCTION

ICU records are frequently and selectively missing, while common imputation evaluations do not directly measure whether pipelines flag the same patients. This paper frames alert-list agreement as a distinct question from prior work on missingness, imputation, and aggregate prediction metrics.

I. INTRODUCTION

  • Selective measurement and sensor disconnection make missingness a routine, potentially informative feature of ICU records, while imputation can replace absence with population-like values or carry earlier observations forward.The cited work notes that observation patterns are associated with disease severity, but that association alone does not distinguish missing-at-random from missing-not-at-random mechanisms,.Source: §I · p. 1
  • Reconstruction error and aggregate discrimination do not directly reveal whether two pipelines flag the same patients, despite alerts being issued to a limited share of patients.The introduction frames the practical question as how many alerts change when only imputation changes, with consequences for outcome studies and equity audits.Source: §I · p. 1

II. RELATED WORK

  • Prior work highlights informative missingness and methods that expose observation patterns to classifiers, while imputation comparisons still use conventional scoring approaches.Missingness indicators alone reached AUROC 0.68 for 30-day mortality in MIMIC-III; the paper omits indicators and time-since-observation features so imputed values remain the varying quantity.Source: §II · p. 2
  • Alert-list concordance differs from prior patient-level metrics by measuring agreement between pipelines over the patients each flags, rather than temporal stability within one patient's risk trajectory.The paper positions calibration, clinical usefulness, and threshold analyses as complementary metrics that remain averaged over patients,.Source: §II · p. 2

III. PROBLEM FORMULATION

The paper treats a deployment alert as a set of patients selected at a fixed alert rate, then compares pipelines by the overlap of those sets. This captures a form of disagreement that AUROC cannot constrain.

III. PROBLEM FORMULATION

  • At a fixed alert rate, alert-list concordance compares the top-k% patient sets induced by two imputation strategies using Jaccard overlap.The Jaccard index is 1 for identical flagged sets and 0 when the sets share no patients.Source: §III · p. 2
  • Two pipelines can have identical AUROC yet share few flagged patients because AUROC depends on positive-negative ordering, not the ordering among non-septic encounters.At 10% alerts and 7.3% patient-level prevalence, non-septic encounters make up roughly seven in ten patients on the list.Source: §III · p. 2
  • The primary comparison evaluates Jaccard across method pairs at alert rates of 5%, 10%, 15%, and 20%, with discrimination and other measures treated as secondary outcomes.The paper also reports rank correlation, calibration, operating-point utility, compute cost, and characteristics of patients on whom strategies disagree.Source: §III · p. 2

IV. METHODS

The experimental pipeline varies only the imputation stage, holding downstream modeling and ranking fixed so differences in alert lists can be attributed to imputation.

IV. METHODS

  • Each condition imputes data, fits one fixed classifier, calibrates predictions, then ranks patients and applies the alert-rate cutoff.Parameters are estimated without touching the test set, and all downstream stages are held constant across conditions.Source: §IV · p. 2

A. Data, Ethics, and Split

The comparison uses a large, highly incomplete ICU cohort and evaluates imputation strategies under shared training constraints. Sequential imputers are implemented at the row level, limiting the scope of what their results characterize.

A. Data, Ethics, and Split

  • The cohort contains 40,336 patients and 1,552,210 patient-hours, with 71.9% average missingness across 39 model features and long gaps between laboratory measurements.Twenty-five of 26 laboratory variables are absent from more than 90% of patient-hours; the median interval between consecutive laboratory measurements is 5 h.Source: A. Data, Ethics, and Split · p. 3
  • All released patients were included and split by patient into 70% training and 30% held-out test sets, with all imputation, classifier, and calibration parameters estimated on training data.The split comprised 28,235 training patients and 12,101 test patients.Source: A. Data, Ethics, and Split · p. 3

B. Imputation Strategies

  • The study compares nine imputers and XGBoostNative, a native-missingness reference, under shared training-only, causal-ordering, and row-level constraints.Strategies span ForwardFill, mean imputation, MICE, autoencoders, and sequential methods including SAITS, BRITS, and GRU-D.Source: B. Imputation Strategies · p. 3
  • Sequential imputers are implemented per patient-hour, so the results characterize them only when paired with a row-level classifier.A full-trajectory imputer with a row-level classifier would capture temporal structure that the classifier then discards.Source: B. Imputation Strategies · p. 3

C. Classifier and Calibration

The classifier configuration and calibration procedure are shared across imputation strategies, while the analysis measures alert-list agreement, discrimination, calibration, operating-point utility, and robustness.

C. Classifier and Calibration

  • The same fixed XGBoost configuration predicts hourly Sepsis-3 labels for every imputed matrix, with no per-method tuning or test-set model selection.The 40th variable orders rows but is excluded as a feature.Source: C. Classifier and Calibration · p. 3
  • Per-patient risk is the maximum calibrated hourly probability, and Platt scaling preserves rankings while making Brier score and ECE calibration-dependent.Calibration used a stratified 20% subsample of training rows, so absolute Brier and ECE values are optimistic.Source: C. Classifier and Calibration · p. 3

D. Outcomes and Analysis

  • Alert-list concordance compares top-k% patient sets across all method pairs, with patient-level bootstrap intervals and rank-correlation measures.The analysis also computes AUROC, AUPRC, Brier score, ECE, precision, recall, and total pipeline time.Source: D. Outcomes and Analysis · p. 3
  • Robustness checks vary alert thresholds, hospital system, and patient-level aggregation, with AUROC and AUPRC differences tested using DeLong’s method.Tests were two-sided and Bonferroni-corrected within comparison families.Source: D. Outcomes and Analysis · p. 3
  • For three method pairs, sepsis-positive encounters are grouped by concordant or discordant alert status, then ICU length of stay is compared across groups.The pairs span ForwardFill-versus-reconstruction Jaccard values of 0.424 and 0.446 and a reconstruction-pair value of 0.606.Source: D. Outcomes and Analysis · pp. 3–4
  • Experiments ran on one V100 GPU, four CPU cores, and 64 GB RAM, with neural imputers on GPU and other methods on CPU.The reported software versions and seed were Python 3.9, XGBoost 1.7.0, scikit-learn 1.0.2, PyTorch 2.0.0, and seed 42.Source: D. Outcomes and Analysis · p. 4

V. RESULTS

Across imputation strategies, aggregate discrimination is similar for many pairs, but their top-decile alert lists can differ; ForwardFill is a persistent overlap outlier.

A. Aggregate Discrimination and Calibration

B. Patient-Level Ranking Stability

  • ForwardFill’s top-decile Jaccard was 0.399 to 0.453, below every other pair; reconstruction-based pairs ranged from 0.502 to 0.750.The native-handling baseline agreed with Mean at 0.670 and with other imputers at 0.465 to 0.543.Source: B. Patient-Level Ranking Stability · p. 4
  • At k = 5%, ForwardFill and MAE had Jaccard 0.305, the widest ForwardFill–reconstruction gap across the tested alert rates.Jaccard broadly increased with k, while the separation between ForwardFill and reconstruction-only pairs persisted from k = 6% to 12%.Source: B. Patient-Level Ranking Stability · p. 4

C. Operating-Point Utility

At matched alert rates, operating-point utility and discrimination provide complementary views of performance, while patient-list overlap and site-wise comparisons reveal pipeline differences.

C. Operating-Point Utility

  • At k = 10%, patient-level precision ranged from 0.257 to 0.298 and recall from 0.353 to 0.410.Across thresholds, precision fell from 0.294–0.371 at k = 5% to 0.207–0.225 at k = 20% as recall rose.Source: C. Operating-Point Utility · p. 4, Table III · p. 5
  • At fixed alert rates, precision compresses the ordinal performance differences, whereas Jaccard measures which patients make up each list.The top-10% pairwise Jaccard matrix is symmetric and displayed as a lower triangle, with ForwardFill, native handling, and reconstruction imputers separated into tiers.Source: C. Operating-Point Utility · p. 5, Figure 1 · p. 5

D. Site-Wise and Patient-Level Robustness

  • The same separation between ForwardFill and other methods appeared within both hospital systems: ForwardFill Jaccard reached at most 0.409 in site A and 0.440 in site B.Among reconstruction methods, only six of 28 patient-level AUROC pairwise comparisons were significant; the other 22 had differences at most 0.011.Source: D. Site-Wise and Patient-Level… · p. 5
  • 22 of 28 reconstruction-method pairs had no significant patient-level AUROC difference after correction, yet alert-list Jaccard was 0.518 to 0.750.The table covers representative pairwise Jaccard values across pooled alert rates and within each hospital system at k = 10%.Source: D. Site-Wise and Patient-Level… · p. 5, Table II · p. 5

E. Sensitivity to the Per-Patient Aggregation Rule

ForwardFill’s low overlap with other methods persists across per-patient aggregation rules, while the length-of-stay pattern depends on the rule. Under a first-24-hour window, the four groups have similar median stays.

E. Sensitivity to the Per-Patient Aggregation Rule

  • Under whole-stay rules, only-ForwardFill patients are long-stay and only-MIDA patients are short-stay; the pattern depends on the aggregation rule.Maximum-over-stay aggregation mechanically favors long stays.Source: E. Sensitivity to the Per-Patient… · p. 5
  • Across all six aggregation rules, ForwardFill Jaccard stays at 0.36–0.46, while reconstruction-only pairs range from 0.45–0.90.Table V reports Jaccard ranges for nine ForwardFill pairs and 36 pairs among non-ForwardFill methods, including XGBoostNative.Source: E. Sensitivity to the Per-Patient… · p. 5, Table V · p. 6
  • With a first-24-hour window, the four groups have similar median stays of 33 to 39 h.The four groups are the only-ForwardFill, only-MIDA, and concordant groups discussed in the section.Source: E. Sensitivity to the Per-Patient… · pp. 5–6

F. Clinical Correlates of Disagreement

Disagreement between methods occurs among encounters with distinct stay lengths and risk profiles, and uniquely flagged patients tend to be alerted later than concordant patients. The hourly pattern in selected ForwardFill cases is consistent with carrying earlier abnormal values through periods without new measurements.

F. Clinical Correlates of Disagreement

  • For ForwardFill comparisons, positive-case Jaccard values were 0.574, 0.570, and 0.722, exceeding the corresponding all-patient values of 0.424, 0.446, and 0.606.Only-ForwardFill patients were alerted a median 5 h before the first labeled hour, versus 22 to 26 h for concordant patients; only-MIDA patients had a 2.5 h median lead.Source: F. Clinical Correlates of Disagreement · p. 6
  • Among sepsis-positive encounters, concordant high-risk groups had the longest stays and concordant low-risk groups the shortest; uniquely ForwardFill-flagged patients had stays near the high-risk group but much lower reconstructed risks.Kruskal-Wallis tests were significant in every panel; Dunn tests placed the concordant low-risk group below every other group in Panels A and B, while only its contrast with the both-flagged group survived correction in Panel C.Source: Table VI · p. 6, F. Clinical Correlates of Disagreement · p. 6
  • The four plotted encounters are an illustrative best case for the mechanism, not a representative sample of the 111 only-ForwardFill encounters.Source: F. Clinical Correlates of Disagreement · p. 6
  • The same clinical signature appeared for the AUROC-indistinguishable MAE–BRITS pair: uniquely flagged patients were alerted later than concordant patients, with median leads of 6.5 h and 1.0 h versus 21 h.Positive-case Jaccard was 0.610 against an all-patient Jaccard of 0.518.Source: F. Clinical Correlates of Disagreement · p. 6
  • In the illustrated encounters, ForwardFill exceeded both reconstruction methods for more than 90% of hours and crossed its alert threshold while SAITS and MICE did not.The pattern is expected if ForwardFill carries an earlier abnormal value through hours without a new measurement.Source: F. Clinical Correlates of Disagreement · p. 6

G. Computational Cost

Across methods with similar discrimination, alert lists can still differ substantially, and the disagreement is concentrated among non-septic patients. Sequential imputers also cost more than Mean on this row-level pipeline, while the interpretation of ForwardFill's distinct alerts remains uncertain.

G. Computational Cost

  • DAE, MAE, and MIDA take 1.5 to 1.9 min, versus 3.2 min for BRITS and 7.5 min for SAITS.These are heterogeneous-hardware wall-clock times; the sequential imputers cost 5 to 12 times Mean despite GPU acceleration.Source: G. Computational Cost · pp. 6–7

VI. DISCUSSION

  • Jaccard 0.50 to 0.75 separates the alert lists of eight strategies within 0.011 AUROC, while 22 pairs are statistically indistinguishable on AUROC.ForwardFill is distinct: it pairs higher discrimination with the matrix's lowest overlap, so its divergence is not equal discrimination producing unequal alerts.Source: §VI · p. 7
  • DAE and MIDA agree most, Mean and XGBoostNative come next, and cross-family pairs agree near 0.50 despite AUROC differences within the 0.011 band.The authors interpret how methods fill missing entries as more relevant to list membership than the small discrimination differences.Source: §VI · p. 7
  • ForwardFill-only patients are alerted a median of 5 h before the first labeled hour, versus 22 to 26 h for concordant patients.Stale extreme values are an alternative explanation, and concordance analysis alone cannot determine whether the unique detections are genuine sensitivity or artifact.Source: §VI · p. 7
  • The finding that deep imputers did not beat Mean applies only to row-level pipelines, which discard temporal dependencies and face 71.9% missingness.Under MNAR, absence itself can be informative, so better reconstruction need not improve discrimination [19].Source: §VI · p. 7
  • Disagreement concentrates among non-septic patients: switching pipelines mostly exchanges false-positive alerts, while sensitivity changes little.Precision varies by at most 0.041; Jaccard measures decision-set stability, not which set is clinically better.Source: §VI · p. 7

VII. LIMITATIONS

The reported conclusions are bounded by the cohort, modeling choices, and feature exclusions. The authors call for external validation and shadow deployment, and note that alert-list changes were not shown to alter clinical decisions or outcomes.

VII. LIMITATIONS

  • Results come from one cohort of two U.S. hospital systems, with models trained on pooled data; transfer to an external cohort remains untested.The site-wise analysis shows the findings hold within each system, but does not resolve the open transfer test.Source: §VII · p. 7
  • SAITS, BRITS, and GRU-D were evaluated in row-level form, not full-trajectory form with a sequential classifier.The authors identify this as the most important limit on what the results cover; MICE used one imputed dataset and MIDA the mean of five draws.Source: §VII · p. 7
  • The study used one learner family without per-method tuning, so tuning each imputer separately could in principle reorder the tiers.Missingness indicators and time-since-observation features were also excluded; their absence may disadvantage imputed pipelines relative to XGBoostNative.Source: §VII · p. 7
  • The study covers one tabular modality and shows who is flagged, not whether escalation decisions or outcomes change.The authors state that shadow deployment is the definitive test.Source: §VII · pp. 7–8

VIII. CONCLUSION

  • AUROC differences were at most 0.011 among 22 reconstruction-method pairs without statistically significant patient-level AUROC differences, while Jaccard ranged from 0.518 to 0.750.ForwardFill separately combined higher discrimination with Jaccard 0.40 to 0.45; the findings held by hospital system and under six patient-level aggregation rules.Source: §VIII · p. 8
Loading 2610.22635v1…