Benchmarking with MIMIC-IV, an irregular, spare clinical time series dataset

1 Introduction

• The study benchmarks algorithms for predicting outcomes from irregular, sparse clinical data using MIMIC-IV.The paper also reviews deep-learning models trained on MIMIC and benchmark and data-pipeline work on MIMIC-III and MIMIC-IV.Source: §1 · pp. 1–2
• MIMIC-IV contains records for 383,220 patients admitted to Beth Israel Deaconess Medical Center ICUs or emergency services from 2008 to 2019.The introduction describes it as considerably larger than MIMIC-III and notes its modular data organization, provenance, and support for using disparate data sources individually or together.Source: §1 · p. 1
• Irregular sampling makes clinical time-series modeling difficult for machine-learning models that assume fully observed, fixed-size feature representations.The paper frames clinical outcome prediction from multivariate time series as relevant to mortality, decompensation, length of stay, and disease risk.Source: §1 · p. 1
• The paper describes an experiment with a training pipeline, models, hyperparameters, and conclusions about training results.It also promises a review of deep-learning models trained with MIMIC and related MIMIC-III benchmark and data-pipeline work.Source: §1 · p. 2

2 Related work

• Existing extraction pipelines address labor-intensive EHR preprocessing, but MIMIC-III tools lack MIMIC-IV support and MIMIC-IV pipelines omit tasks or offer limited model choices.COP-E-CAT excludes non-ICU data, readmission, and LOS; Gupta et al. (2022) provides end-to-end modules but supports a limited number of predictive models; Liao and Voldman (2023) omits LOS.Source: §2 · p. 2
• mTAN and HeTVAE use multi-time attention for irregular series, but address sparsity and uncertainty differently and both omit most MIMIC features and static demographics.mTAN uses a homoscedastic output and normalizes away input sparsity; HeTVAE introduces uncertainty-aware UnTAN to encode input uncertainty more directly.Source: §2 · p. 2
• STraTS represents time-series as observation triplets without imputation or discretization, while TCNs use dilations, residual connections, and causal convolutions; reported MIMIC-III results favor TCNs over generic recurrent models.STraTS uses Continuous Value Embedding and transformer attention; Bednarski et al. (2022) claimed TCNs substantially outperform LSTMs and GRUs on MIMIC-III.Source: §2 · p. 2
• Other methods adapt recurrent models to irregular gaps: T-LSTM adjusts hidden states, ODE-RNN models continuous-time dynamics, and GRU-D and DATA-GRU decay hidden states over unobserved intervals.GRU-D also decays inputs toward global means, while DATA-GRU adds dual attention to preprocess training data.Source: §2 · pp. 2–3
• On MIMIC-III, Purushotham et al. (2018) used first-24/48-hour inputs and LOS regression (mean squared error), while Harutyunyan et al. (2019) predicted LOS hourly as classification (Cohen’s kappa); the authors call the algorithms outdated.The studies used different data extraction, preprocessing, and evaluation metrics, motivating a standardized pipeline.Source: §2 · p. 3

3.1 MIMIC-IV dataset

• The experiments use MIMIC-IV 2.2, released in January 2023, whose hospital-wide and ICU-specific data are separated into modules and include both ICD-9 and ICD-10 codes.MIMIC-IV covers 2008–2019, whereas MIMIC-III covers 2001–2012; the dataset passage reports 209,359 admissions with ICD-9 codes and 122,317 with ICD-10 codes.Source: §3.1 · p. 3

3.2 Data preparation

• The study uses ICU-only MIMIC-IV 2.2 data for mortality and length-of-stay prediction, evaluating models with ROC-AUC and PR-AUC.It adapts Gupta et al. (2022)’s extensible extraction and preprocessing pipeline; admission prediction is excluded because it requires additional text features and large-language-model training.Source: §3.2 · p. 4
• Data extraction is restricted to ICU admissions for chronic kidney disease, using diagnosis and labs/vitals features; ICD-9 diagnosis codes are converted to ICD-10 before clinical grouping.The features include 407 labs/vitals and 1,034 diagnosis features for mortality, and 408 labs/vitals and 1,117 diagnosis features for length of stay; the respective training sets contain 7,910 and 12,604 admissions.Source: §3.2 · pp. 4–5

3.4 Result

• XGBoost achieves the best scores on both prediction tasks; LSTM outperforms TCN for mortality, while TCN outperforms LSTM for length of stay.The comparison uses five-fold cross-validation with an 80% training and 20% test split; deep-learning training further reserves 10% of training data for validation each epoch.Source: Table 3 · p. 5, §3.4 · p. 5

3.3 Prediction Algorithms

• The prediction algorithms compared are XGBoost, a traditional machine-learning model, and two deep-learning models: LSTM and TCN.Table 2 summarizes the deep-learning models’ hyperparameters.Source: §3.3 · p. 5

• The study provides a standardized benchmark for MIMIC-IV, described as one of the most popular EHR datasets.Source: §4 · p. 5
• Traditional models such as XGBoost can outperform complex deep learning algorithms and achieve good results on mortality and length-of-stay prediction.Source: §4 · pp. 5–6
• The authors plan to update the benchmark with more algorithms, especially state-of-the-art methods, and add prediction tasks such as phenotyping and admission.Source: §4 · p. 6