PaperlayerView exact revision on arXiv

Source-linked AI summary

Feature Robustness in Non-stationary Health Records: Caveats to Deployable Model Performance in Common Clinical Machine Learning Tasks

Bret Nestor, Matthew B. A. McDermott, Willie Boag, Gabriela Berner, Tristan Naumann, Michael C. Hughes, Anna Goldenberg, Marzyeh Ghassemi

arXiv:1908.00690v1cs.LGstat.ML

TL;DR

De-identification leaves common MIMIC-III evaluations unable to assess historical-to-future generalization. The authors use year-of-care access to test temporal drift and find that expert-defined clinical aggregation reduces AUROC losses across mortality and length-of-stay prediction.

  • Problem

    Because MIMIC-III de-identification obscures care dates, common time-agnostic evaluation cannot assess how models trained on historical data generalize to future data.

    Source: §1 · p. 2

  • Method

    The authors use restricted year-of-care access to evaluate models across future years and introduce expert-defined clinical groupings of raw features.

    Source: §1 · pp. 2–3

  • Results

    Maximum AUROC loss falls from 0.29 to 0.06 for ICU mortality and from 0.10 to 0.03 for length of stay with aggregate representations.

    Source: §7 · p. 12

  • Takeaways & Limitations

    Time-agnostic evaluations overestimate prediction quality, while expert-defined aggregate representations generalize to future years better than several automated preprocessing methods.

    Source: §7 · p. 12

    The aggregate representations were manually defined for MIMIC, so transfer to other EHR systems requires new taxonomies developed with clinical experts.

    Source: §7 · p. 12

Abstract

from arXiv · show

When training clinical prediction models from electronic health records (EHRs), a key concern should be a model's ability to sustain performance over time when deployed, even as care practices, database systems, and population demographics evolve. Due to de-identification requirements, however, current experimental practices for public EHR benchmarks (such as the MIMIC-III critical care dataset) are time agnostic, assigning care records to train or test sets without regard for the actual dates of care. As a result, current benchmarks cannot assess how well models trained on one year generalise to another. In this work, we obtain a Limited Data Use Agreement to access year of care for each record in MIMIC and show that all tested state-of-the-art models decay in prediction quality when trained on historical data and tested on future data, particularly in response to a system-wide record-keeping change in 2008 (0.29 drop in AUROC for mortality prediction, 0.10 drop in AUROC for length-of-stay prediction with a random forest classifier). We further develop a simple yet effective mitigation strategy: by aggregating raw features into expert-defined clinical concepts, we see only a 0.06 drop in AUROC for mortality prediction and a 0.03 drop in AUROC for length-of-stay prediction. We demonstrate that this aggregation strategy outperforms other automatic feature preprocessing techniques aimed at increasing robustness to data drift. We release our aggregated representations and code to encourage more deployable clinical prediction models.

1. Introduction

Because de-identification obscures care dates, common MIMIC-III evaluation cannot test historical-to-future generalization. Using restricted year-of-care access, the authors find substantial performance losses under temporal drift and propose expert-defined clinical aggregation as a mitigation.

1. Introduction

  • De-identification makes MIMIC-III temporally inconsistent, preventing historical-to-future evaluation of care-practice drift, including the 2008 change in record-keeping systems.Source: §1 · p. 2
  • Using a Limited Data Use Agreement, the study evaluates how multiple models generalize to unseen future data across input representations and time-aware training regimes.Source: §1 · p. 2
  • Raw models show worst-case AUROC drops of 0.29 for RF mortality and 0.10 for RF long-LOS; clinical aggregation reduces these drops to 0.06 and 0.03, respectively.Source: §1 · pp. 2–3
  • The authors report that mortality and length-of-stay prediction quality saturates quickly with little data, suggesting focus on more difficult tasks.Source: §1 · p. 3
  • The paper profiles models across representations and temporal training regimes, and releases expert-mapped clinical buckets as a resource for future work.Source: §1 · p. 3

2. Outline

The paper evaluates two binary ICU prediction tasks using combinations of feature representations, models, and training regimes, then situates the work among research on generalization across time and clinical sites.

2. Outline

  • The study focuses on mortality and long length-of-stay prediction, evaluating permutations of feature representations, prediction models, and training paradigms.Source: §2 · p. 3
  • Representations range from raw features through automatic preprocessing to expert-selected high-level clinical features.Source: §2 · p. 3
  • The paper reviews related work on generalizing across time and clinical sites after reporting its results.Source: §2 · p. 3

3. Data Cohort and Prediction Tasks

The study builds a first-ICU-stay cohort, represents the first day of measurements as hourly features, and predicts ICU mortality or stays longer than three days.

3.1. Data Cohort

  • The cohort contains 21,877 unique ICU stays from patients over 15, restricted to first ICU visits lasting at least 36 hours.Source: §3.1 · p. 4

3.2. Features: Demographics and Hourly Labs and Vitals.

  • Each stay includes seven demographic features and 181 labs and vital measurements; hourly aggregation, training-fitted normalization, 24-hour censoring, and simple imputation prepare the inputs.Source: §3.2 · pp. 4–5

3.3. Binary Prediction Tasks

  • Using the first 24 hours, the tasks predict ICU death or a stay longer than three days, with at least a 12-hour gap before the predicted event.Source: §3.3 · p. 5

4. Methods

The methods compare four feature representations and four model types under time-aware evaluation regimes, using the first-day ICU time series as the prediction input.

4. Methods

  • The pipeline assesses representations, prediction methods, and chronologically consistent train-test divisions to approximate prospective deployment.Source: §4 · p. 5

4.1. Representations

  • Raw ItemIDs are tied to EHR software and can change across system transitions; alternatives include PCA, ontology-based CUI spanning, and expert-defined clinical aggregations.Source: §4.1 · pp. 5–6

4.2. Models

  • The four evaluated models are logistic regression, random forests, LSTM, and GRU-D; recurrent models process time series directly, while LR and RF use flattened series.Source: §4.2 · pp. 6–7

4.3. Training Regimes

  • Training regimes compare fixed 2001–2002 training, prior-year-only training, and all-available-prior-history training for future-year tests.Source: Figure 2 · p. 7, §4.3 · pp. 7–8

5. Results

Clinical aggregation is more robust than raw and automated representations during temporal change, while date-agnostic evaluation overstates performance. Under clinical aggregation, performance on future data is already steady with one year of training data.

5.1. Clinical Aggregate Representations are Most Robust to Non-stationarity

  • Clinical aggregation maintains more consistent performance across years and lower standard deviation than raw features, as shown in the full-history mortality comparison.The table compares average AUROC and maximum loss across unseen years; its caption reports that clinical representations tend to improve overall performance and reduce deterioration.Source: §5.1 · p. 8, §5 · p. 8, Table 1 · p. 9
  • PCA performs below raw features over time, while automatic groupings help some models but do not restore random forest or LSTM performance.The partial benefit from automatic grouping is reported for GRU-D and logistic regression; PCA also struggles to generalise throughout time.Source: §5.1 · p. 8
  • Smaller protected subgroups generally show higher variance and worse generalisation than larger subgroups.The paper describes this subgroup pattern as generally observed and points to Appendix F for further information.Source: §5.1 · p. 8

5.2. Date-Agnostic Training Overstates Performance, Especially in Raw Data

  • 0.82 ± 0.02 year-agnostic mortality AUROC versus 0.76 ± 0.13 year-averaged AUROC for raw-representation random forests; clinical aggregation yields 0.86 ± 0.02 versus 0.85 ± 0.02.The reported overestimate is 0.054 for raw features and 0.017 for clinical aggregation.Source: §5.2 · pp. 8–9
  • Lower-capacity random forests and logistic regression are preferred on both tasks in the non-stationary setting.The paper relates this pattern to prior work reporting that model-complexity gains under stationary conditions disappear in prospective non-stationary evaluation.Source: §5.2 · p. 9

5.3. Performance Saturation and Task Importance

  • With clinical aggregates, future-year performance is steady from one training year even as the dataset grows tenfold; the figure marks the CareVue-to-MetaVision transition.The paper suggests focusing on harder tasks or signals beyond the measured 181 labs and vitals.Source: §5.3 · p. 9, Figure 3 · p. 10

6. Related Work

Prior work studies concept drift in specific settings and transfer across institutions, while much MIMIC-III benchmarking reports performance on date-agnostic splits. The paper’s related work distinguishes temporal robustness from institution-to-institution generalisability.

Machine Learning for Health on MIMIC

  • MIMIC-III benchmarks cover many clinical prediction tasks but commonly report aggregate performance on date-agnostic train/test splits.Prior MIMIC-III work defines cohorts and benchmark tasks to compare models of varying capacity.Source: §6 · p. 11

Robust Machine Learning over Time

  • Concept-drift research spans isolated healthcare settings, including kidney-injury and surgery prediction, while proposed approaches address drift or dataset shift in different ways.The cited work reports maintained predictive performance but calibration drift in one setting, and proposes detection or causal-feature approaches in others.Source: §6 · p. 11

Multi-site Generalisability

  • Prior cross-institution approaches use CUI-based aggregation or site-agnostic representations, whereas this work examines abrupt EHR-system changes with little feature overlap.Gong et al. (2017) treats the CareVue-to-MetaVision transition as a proxy for multiple institutions rather than temporal drift; other approaches assume shared systems or enough samples to capture sparse-feature correlations.Source: §6 · pp. 11–12

7. Conclusion

Using year-of-care data, the study evaluates temporal robustness on mortality and length-of-stay prediction and finds that time-agnostic evaluation overestimates performance. It reports smaller maximum AUROC losses with its aggregate representations, while noting limits in data access, transfer, and task coverage.

7. Conclusion

  • Maximum AUROC loss falls from 0.29 to 0.06 for ICU mortality and from 0.10 to 0.03 for length of stay with the aggregate representations.The comparison covers MIMIC-III from 2002–2012; the study also reports that aggregate representations outperform several automated preprocessing methods.Source: §7 · p. 12
  • The study covers only two MIMIC-III tasks, and its manually defined aggregate representations would require new expert-developed taxonomies for other EHR systems.Access to dated events through a Limited Use agreement may not be immediately available to others.Source: §7 · p. 12
  • The authors caution that models trained on date-agnostic, de-identified data should not be expected to translate into clinical practice.They recommend considering deployability limits when benchmarking clinical machine learning progress.Source: §7 · pp. 12–13

Appendix A. Model Training Details

Appendix A describes model training and tuning, including the decision not to vary imputation schemes; Appendix B gives feature-aggregation examples and illustrates changes in collection frequency and measured values.

Appendix A. Model Training Details

  • Imputation schemes were not varied, allowing the study to isolate the effect of representation on generalisability.The authors note that imputation effects on model performance were not explicitly tested.Source: Appendix A · p. 18
  • LR and RF use 24 one-hour history buckets concatenated with demographics; LR is linear, while RF uses bagged decision trees.Both model descriptions specify SciKit Learn implementations.Source: Appendix A · p. 18
  • The bidirectional LSTM processes sequences, while GRU-D models irregular sampling; GRU-D was not evaluated on PCA because that representation absorbs missingness.The LSTM description lists random-search hyperparameters; the GRU-D description specifies a 67-unit hidden layer, batch normalization, dropout, Adam, and early stopping.Source: Appendix A · p. 18
  • For LR, RF, and LSTM, five-fold cross-validation and random search selected parameters maximizing validation AUROC; GRU-D used validation-based early stopping.The reported GRU-D variance reflects only the train/validation split.Source: Appendix A · p. 19

Appendix B. Feature aggregation description

  • The aggregation example averages CHARTEVENTS ItemIDs 861, 1127, 1542, and 220546 into one White blood cell count feature.The first three ItemIDs belong to the 2001–2008 system and 220546 to the 2008–2012 system.Source: Appendix B · p. 19
  • Figures 4 and 5 illustrate that collection frequency and measured values can change in clinical practice.Figure 4 uses Mean Arterial Blood Pressure as its collection-frequency example.Source: Figure 4 · p. 19, Figure 5 · p. 19

Appendix C. Task Descriptions

The appendix defines in-ICU mortality and long length of stay as binary prediction tasks, giving their cohort rates and task-specific context.

Appendix C. Task Descriptions

  • In-ICU mortality is patient death within the ICU, and the cohort’s ICU mortality rate is 7.4%.The passage also notes that mortality is a common prediction target and a direct acuity signal strongly associated with EHR signals.Source: Appendix C · p. 19
  • The study defines long length of stay as a stay greater than the cohort median of 3 days, yielding a 47.1% positive subject rate.The task predicts whether a patient’s ICU stay is greater than or less than 3 days.Source: Appendix C · p. 20
  • The long-LOS task provides a more balanced-class setting without a directly targeted policy change.Source: Appendix C · p. 20

Appendix D. Year Agnostic Results

The year-agnostic results are presented as the conventional reporting setup, with AUROC summaries for in-ICU mortality and long-LOS classification.

Appendix D. Year Agnostic Results

  • Tables 2 and 3 report model performance trained without knowledge of the years, using 5 × 2-fold cross-validation splits.Source: Appendix D · p. 20
  • Table 2 reports mean ± standard deviation AUROC for year-agnostic in-ICU mortality performance.Source: Table 2 · p. 20
  • Table 3 reports mean ± standard deviation AUROC for year-agnostic classification of stays longer than 3 days.Source: Table 3 · p. 20

Appendix E. Performance on Length-of-Stay Task

The length-of-stay results show performance decay around the CareVue-to-MetaVision shift, with clinical aggregation recovering performance; the appendix compares raw and aggregated representations under Full History training.

Appendix E. Performance on Length-of-Stay Task

  • The appendix presents the full length-of-stay results reported in the main text.Source: Appendix E · p. 20
  • Clinical aggregation recovers length-of-stay performance after the CareVue-to-MetaVision shift, while insufficient-prior-data training regimes show year-over-year decay.Figure 6 shows AUROC over time across representations and training regimes.Source: Appendix E · pp. 20–21, Figure 6 · p. 21
  • Table 4 compares average AUROC and maximum AUROC drop for Raw versus Clinical Aggregation representations under Full History training.Its averages cover unseen years from 2003 onward; maximum drop compares the first evaluation year with subsequent unseen years.Source: Appendix E · p. 21, Table 4 · p. 22

Appendix F. Subpopulation Sensitivity Analyses

Appendix F examines mortality-prediction performance across gender, ethnicity, and insurance subpopulations. Results are erratic for groups with fewer samples, and deterioration differs between private-insurance and Medicaid patients.

Appendix F. Subpopulation Sensitivity Analyses

  • Mortality results were highly erratic for self-pay and government-insurance patients, which the authors attribute to few training samples.Source: Appendix F · p. 23
  • Models deteriorated more slowly for patients with private insurance than for patients with Medicaid.Source: Appendix F · p. 23
  • Insurance type was absent from the feature sets; Figure 11 shows GRU-D mortality performance by highlighted insurance types, while Figure 10 shows annual ICU admissions by insurance type.Source: Appendix F · p. 23, Figure 11 · p. 24, Figure 10 · p. 24
  • The ethnicity analysis pairs mortality performance for highlighted demographics in Figure 9 with annual ICU admission counts by ethnicity in Figure 8.Source: Appendix F · p. 23, Figure 8 · p. 23, Figure 9 · p. 24
  • Figure 7 examines how representation affects GRU-D in-ICU mortality performance longevity across two genders, with error bars showing ± standard error.Source: Figure 7 · p. 23, Appendix F · p. 23
Loading 1908.00690v1…