PaperlayerView exact revision on arXiv

Source-linked AI summary

Multitask learning and benchmarking with clinical time series data

Hrayr Harutyunyan, Hrant Khachatrian, David C. Kale, Greg Ver Steeg, Aram Galstyan

arXiv:1703.07771v3stat.MLcs.LG

TL;DR

Clinical machine-learning progress is difficult to measure without public benchmarks, while researchers often study interrelated prediction tasks separately. This paper introduces four MIMIC-III clinical prediction benchmarks and finds that LSTM-based models significantly outperform linear models, with recurrent networks exploiting task correlations to improve several tasks.

  • Problem

    Clinical machine-learning progress lacks publicly available benchmark datasets, and interrelated prediction tasks are often studied separately despite simultaneous clinical use.

  • Method

    The paper constructs four standardized clinical prediction benchmarks from multivariate time series and labels covering mortality, decompensation, length of stay, and phenotype classification in MIMIC-III.

  • Results

    LSTM-based models significantly outperform linear models, while carefully designed recurrent networks exploit task correlations to improve performance on several tasks.

  • Takeaways & Limitations

    The benchmark suite is immediately accessible for replication and further research on standardized clinical prediction tasks.

  • Takeaways & Limitations

    Because MIMIC-III data come from a single EHR system, models trained on these benchmarks may contain systematic biases and require evaluation on other clinical datasets.

Abstract

from arXiv · show

Health care is one of the most exciting frontiers in data mining and machine learning. Successful adoption of electronic health records (EHRs) created an explosion in digital clinical data available for analysis, but progress in machine learning for healthcare research has been difficult to measure because of the absence of publicly available benchmark data sets. To address this problem, we propose four clinical prediction benchmarks using data derived from the publicly available Medical Information Mart for Intensive Care (MIMIC-III) database. These tasks cover a range of clinical problems including modeling risk of mortality, forecasting length of stay, detecting physiologic decline, and phenotype classification. We propose strong linear and neural baselines for all four tasks and evaluate the effect of deep supervision, multitask training and data-specific architectural modifications on the performance of neural models.

Introduction

The paper addresses stalled progress in clinical machine learning by introducing reproducible public benchmarks from MIMIC-III and framing four prediction tasks as a heterogeneous multitask problem. It shows that carefully designed recurrent neural networks can exploit temporal task correlations to improve performance on several tasks.

  • Motivation: Electronic health records create opportunities for data mining on problems including triage, risk assessment, physiologic decompensation, high-cost patients, and complex diseases.Over 30 million patients visit U.S. hospitals annually, and 83% use an EHR system.
  • Problem: The absence of widely accepted benchmarks slows clinical machine learning progress by limiting evaluation, reproducibility, competition, and community focus.The paper identifies benchmarking as the main challenge in harnessing digital health data.
  • Problem: Researchers often develop methods for individual prediction tasks even though clinical staff perform related tasks simultaneously in decision making.The paper also notes accumulating evidence that these prediction tasks are interrelated.
  • Contribution: The proposed public benchmark suite uses MIMIC-III to cover in-hospital mortality, physiologic decompensation, length of stay, and phenotype classification.The benchmark contains rich clinical data and is designed around four clinical prediction tasks.
  • Method: The benchmarks define a heterogeneous multitask learning problem because the four tasks differ in output type and temporal structure.Length of stay requires regression at each time step, whereas in-hospital mortality risk is predicted once early in admission.
  • Contribution: Carefully designed recurrent neural networks handle sequence data, model correlations between tasks distributed in time, and improve performance for several tasks.The code is available online so MIMIC-III users can reproduce the benchmarks and experiments without repeating the preprocessing difficulties.

Related Work

Prior clinical prediction research established strong neural performance across mortality, length-of-stay, and phenotyping tasks, while multitask work had not addressed task-specific temporal structure. Existing MIMIC-III benchmarks also differed from this work in their input windows and task coverage.

  • Clinical prediction: Feedforward neural networks nearly always outperform logistic regression and severity of illness scores for hospitalized-patient mortality risk modeling.LSTM-based architectures have also performed well on inpatient mortality, 30-day unplanned readmission, long length-of-stay classification, and diagnosis prediction using private general-EHR datasets.
  • Length of stay: Earlier length-of-stay research commonly framed prediction as binary classification for long-stay risk rather than regression.Recent deep learning architectures have also been proposed for survival analysis, a related time-to-event regression task with right censoring.
  • Phenotyping: Deep learning phenotyping studies have used feedforward, LSTM, and temporal convolutional networks to predict diagnostic codes from clinical time series.Recurrent neural networks were also shown to classify dozens of acute care diagnoses in variable-length clinical time series.
  • Multitask learning: Multitask clinical prediction work included multi-label phenotyping and joint mortality–length-of-stay prediction, but did not address settings where sequential structure varies across tasks.The paper identifies this gap as the distinction between prior multitask approaches and its own work.
  • MIMIC-III benchmarks: A parallel MIMIC-III benchmark included mortality, length-of-stay, and ICD-9 group prediction but omitted decompensation prediction.Its tasks used only the first 24 or 48 hours of data, whereas this work predicts length of stay and decompensation hourly and performs phenotyping using data of

1. In-hospital mortality

The in-hospital mortality benchmark predicts binary mortality from the first 48 hours of an ICU stay, evaluated primarily with AUC-ROC. A related decompensation task predicts next-24-hour mortality hourly during an ICU stay as a proxy for rapid health deterioration.

  • In-hospital mortality: In-hospital mortality is a binary classification task predicting mortality from the first 48 hours of an ICU stay.Its main evaluation metric is area under the receiver operating characteristic (AUC-ROC).
  • Decompensation: Decompensation predicts whether a patient will rapidly deteriorate within the next 24 hours, aiming to replace hospital early warning scores.Because no gold standard exists for evaluating early warning scores, the benchmark defines decompensation as mortality prediction in the next 24 hours at each ICU-stay hour.

2. Decompensation prediction

The task predicts the remaining time a patient will spend in the ICU at each hour of their stay. It is framed as 10-class classification and evaluated primarily with Cohen’s linear weighted kappa.

  • Task definition: The benchmark predicts remaining ICU length of stay at each hour of a patient’s stay.Accurate prediction supports scheduling and hospital resource management.
  • Task definition: The task uses 10 length-of-stay classes: stays shorter than a day, seven daily buckets for the first week, one 1–2-week bucket, and one bucket for stays over two weeks.
  • Evaluation: The main evaluation metric is Cohen’s linear weighted kappa score.

3. Length-of-stay prediction

Length-of-stay prediction compared multiclass classification with direct regression of days, finding that regression traded lower kappa for better mean absolute difference. The best multiclass model achieved an AUC-ROC of 0.84 for predicting stays longer than seven days from the first 24 hours.

  • Length-of-stay prediction: LSTM-based models outperformed linear models by substantial margins across all metrics on every task, including length-of-stay prediction.The reported comparison covered linear models and LSTM-based neural architectures.
  • Length-of-stay prediction: Channel-wise LSTMs significantly outperformed standard LSTMs for length-of-stay prediction, while multitask training significantly improved the task.Deep supervision significantly improved length-of-stay prediction except for the standard LSTM model.
  • Length-of-stay prediction: Regression models predicting days consistently had worse kappa but better mean absolute difference than classification models.They were trained with mean squared error, so their MAD scores were considered suboptimal.
  • Length-of-stay prediction: 0.84 AUC-ROC was achieved by the best multiclass LOS model for predicting stays longer than seven days at 24 hours after admission.The score was obtained by summing predicted probabilities for buckets representing LOS of seven days or longer.

Phenotyping

Phenotyping comprises 25 binary classification tasks with substantial variation in performance across diseases. Acute phenotypes are easier to predict than chronic ones, and ROC-AUC is not positively correlated with disease prevalence.

  • Phenotyping: Phenotype ROC-AUC ranges from 0.6834 for essential hypertension to 0.9089 for acute cerebrovascular disease.The channel-wise LSTM is the best phenotype baseline.
  • Phenotyping: Chronic diseases have lower ROC-AUC than acute diseases, at 0.7475 versus 0.7964.
  • Phenotyping: No positive correlation is detected between disease prevalence and ROC-AUC, with essential hypertension showing the worst performance despite being the most common phenotype.

Multitask learning

The proposed multitask architecture extracted useful information unavailable to single-task models, improving performance in some settings, but offered no significant phenotyping benefit. Training remains challenging because the four tasks converge and overfit at different rates.

  • Multitask learning: Multitask LSTM leveraged useful input-sequence information that single-task models could not, explaining better performance in some settings.The authors found no significant multitask-learning benefit for phenotyping.
  • Multitask learning: The four tasks converged and overfit at different rates, complicating multitask training and motivating heuristics such as per-task early stopping.Per-task loss weighting reduced the problem, but the supplied passage ends before stating the complete outcome.
  • Multitask learning: The section covers benchmark data and task generation, evaluation metrics, linear and neural baselines, experimental setup, and model selection.These topics are organized into three subsections.

Benchmark tasks

The benchmarks comprise four in-hospital clinical prediction tasks spanning mortality risk, physiologic decompensation, length-of-stay forecasting, and phenotype classification. The mortality benchmark uses observations from the first 48 hours of an ICU admission and labels whether the patient died before hospital discharge.

  • Benchmark tasks: The benchmark data are derived from MIMIC-III and the processed cohort contains over 31 million events from 42,276 ICU stays.The preparation workflow extracts, filters, and cleans clinical events before constructing task-specific data sets.
  • Benchmark tasks: The benchmark covers four in-hospital tasks: mortality risk, physiologic decompensation detection, continuous length-of-stay forecasting, and phenotype classification.These tasks span multiple clinical problems and machine-learning task types.
  • In-hospital mortality: Mortality prediction uses observations recorded early in an ICU admission to estimate whether a patient dies before hospital discharge.The benchmark uses a wider 48-hour observation window than the typical 12–24-hour window.
  • In-hospital mortality: Mortality performance is reported using AUC-ROC and AUC-PR, with AUC-PR included because it can be more informative for highly skewed data sets.AUC-ROC is the most commonly reported metric in mortality prediction research.
  • In-hospital mortality: The mortality data set contains 17,903 training and 3,236 test ICU stays, with a 13.23% mortality rate.The mortality rate corresponds to 2,797 of 21,139 ICU stays.

Baselines

The benchmarks evaluate hand-engineered logistic regression and LSTM-based neural-network baselines across four clinical prediction tasks. Neural baselines include standard and channel-wise LSTMs, deep supervision, and multitask variants using other tasks as auxiliary objectives.

  • Baseline families: The baseline models comprise logistic regression with hand-engineered features and LSTM-based neural networks, evaluated on all four benchmark tasks.Both model families have been effective for clinical prediction from physiologic time series.
  • Logistic regression: Logistic regression summarizes each variable with six statistics across seven time-series subsequences, including the full series and temporal prefixes and suffixes.The statistics are minimum, maximum, mean, standard deviation, skew, and number of measurements.
  • Standard LSTM: The Standard LSTM processes concatenated inputs with a single LSTM layer and task-specific output layers and loss functions.Inputs are regularly resampled, missing values are imputed, and binary observation masks are provided for each variable.
  • Channel-wise LSTM: The channel-wise LSTM independently processes different variables with separate bidirectional LSTM layers, concatenates their outputs, and feeds them to another LSTM layer.Its output layers and loss functions match those of the standard LSTM, while its design preserves variable-specific information before mixing channels.
  • Deep supervision: Deep supervision predicts targets at each time step, using target replication for mortality and phenotype tasks and grouped-instance supervision for decompensation and length-of-stay tasks.For decompensation and length of stay, target replication is unsuitable because the final-step target may be incorrect for earlier steps.
  • Multitask learning: Multitask baselines use the other three tasks as auxiliary tasks and can be implemented with either standard or channel-wise LSTMs.The proposal is motivated by correlations between targets across tasks.

Experiments, Model selection and Evaluation

Models were selected by validation performance using grid search, with final evaluation on sparingly used test sets. Experiments specified task-dependent LSTM settings and multitask loss weights, while bootstrapping estimated confidence intervals for test scores.

  • Model selection: Validation used the same predefined 15% of patients for all algorithms, with models trained on the remaining 85% and final scores reported on test sets.Hyperparameters were tuned by grid search, and test sets were used sparingly during development to avoid unintentional leakage.
  • LSTM configuration: One-hour intervals were used for LSTM discretization because they balanced missingness against repeated measurements and matched decompensation and LOS prediction sampling.Using 0.8-hour intervals produced no improvement.
  • LSTM configuration: Dropout reduced overfitting, and all LSTM mortality baselines used 30% dropout because overfitting was most severe for that task.The best LSTM hyperparameters varied across tasks and were listed in the repository’s pretrained_models.md file.
  • Multitask model selection: Multitask overfitting degraded mortality and decompensation validation performance faster than other tasks, with best loss weights differing by task.(1,0.2,1.5,1) was best for mortality, decompensation, and length of stay, whereas (0.1,0.1,0.5,1) was better for phenotype prediction.
  • Evaluation: Test-set confidence intervals were estimated by bootstrapping because test scores estimate performance on unseen examples.The paper describes bootstrapping as a method for estimating measure variability, testing model differences, and reporting 95% confidence intervals.
Loading 1703.07771v3…