PaperlayerView exact revision on arXiv

Source-linked AI summary

Improving Clinical Predictions through Unsupervised Time Series Representation Learning

Xinrui Lyu, Matthias Hueser, Stephanie L. Hyland, George Zerveas, Gunnar Raetsch

arXiv:1812.00490v1cs.LGstat.ML

TL;DR

Limited expert annotation and varied clinical tasks motivate learning useful representations from unlabeled medical time series. The paper evaluates Seq2Seq autoencoders and forecasters, including an attention-based forecaster, and reports that unsupervised representations outperform end-to-end supervised models in the examined data-scarce setting.

  • Problem

    Medical time series are plentiful, but expert annotation is limited and clinical tasks vary, motivating representations useful across downstream tasks.

    Source: §1 · p. 1

  • Method

    The paper trains Seq2Seq models unsupervised as either input-reconstructing autoencoders or future-signal forecasters, including a forecaster with attention.

    Source: §3.1 · p. 2, §3.3 · p. 3

  • Results

    In the examined data-scarce regime, classifiers using unsupervised representations outperform end-to-end supervised models, and S2S-F-A has the highest performance in future signal and clinical outcome prediction among the studied models.

    Source: §4 · p. 4, §5 · p. 5

  • Takeaways & Limitations

    Unsupervised representations can provide performance gains over supervised learning from raw signals when labeled medical time series data is scarce.

    Source: §5 · p. 5

Abstract

from arXiv · show

In this work, we investigate unsupervised representation learning on medical time series, which bears the promise of leveraging copious amounts of existing unlabeled data in order to eventually assist clinical decision making. By evaluating on the prediction of clinically relevant outcomes, we show that in a practical setting, unsupervised representation learning can offer clear performance benefits over end-to-end supervised architectures. We experiment with using sequence-to-sequence (Seq2Seq) models in two different ways, as an autoencoder and as a forecaster, and show that the best performance is achieved by a forecasting Seq2Seq model with an integrated attention mechanism, proposed here for the first time in the setting of unsupervised learning for medical time series.

1 Introduction

Medical time series are abundant, but expert annotation is limited and clinical tasks vary across settings. The paper investigates unsupervised representations intended to perform across clinically relevant prediction tasks.

1 Introduction

  • Routine practice produces abundant patient time series, while expert annotation is limited and new tasks and task definitions vary across health systems.These conditions motivate a need for representations that work across a broad range of downstream tasks.Source: §1 · p. 1
  • The paper studies unsupervised representation learning on medical time series and evaluates when it benefits clinically relevant outcome prediction over end-to-end supervised learning.Source: §1 · p. 1

2 Related Work

Prior work established autoencoding and Seq2Seq approaches, and medical applications of unsupervised representations. The paper addresses limited study of reconstruction objectives for clinical outcome prediction and attention-based medical representations.

2 Related Work

  • Autoencoding has variants including denoising, variational, convolutional, and contractive methods, while Seq2Seq models have been used in language and image tasks and unsupervised pretraining.Source: §2 · p. 1
  • Medical unsupervised representations have shown descriptive and predictive utility, but reconstruction principles and losses for accurate clinical outcome prediction remain understudied.Source: §2 · p. 1
  • Attention has broad use and can improve performance and interpretability, but medical attention architectures had focused on specific supervised tasks.Source: §2 · p. 2

3 Representation Learning Models

The paper compares flattened non-sequential autoencoder baselines with sequential models trained to reconstruct or forecast patient time series. It adds attention to the forecasting model while retaining the encoder’s final state as the downstream representation.

3.1 Baselines: Autoencoders

  • PCA and multilayer-perceptron autoencoders process flattened time series, so they do not exploit temporal structure in the original sequence.Each T-by-d sequence is flattened to a Td-dimensional vector for these baselines.Source: §3.1 · p. 2
  • The S2S-AE uses LSTMs to encode a window into its final hidden state and decode that representation to reconstruct the same input window.Its training objective minimizes reconstruction error, and the representation summarizes a fixed-length medical history.Source: §3.1 · p. 2

3.2 Sequential forecasting model (S2S-F)

  • S2S-F changes the decoder target from the input window to the future time series, optimizing representations to predict future patient state.The model forecasts the next T time points from the encoded history.Source: §3.2 · p. 3

3.3 Forecasting with attention (S2S-F-A)

  • S2S-F-A adds attention to forecasting so the decoder can weight input-sequence states according to their relevance at each output time.The resulting context vector resummarizes the input for the current decoding task and is concatenated to the decoder input; the downstream representation remains the encoder’s final state.Source: §3.3 · p. 3

4 Experiments and results

In the tested clinical prediction setting, unsupervised time-series representations outperform end-to-end supervised models when labels are scarce. Forecasting representations, especially with attention, also improve future-signal prediction.

Data

  • Experiments used eICU data from 20,878 patients, with 94 hourly variables and histories spanning 72–720 hours.The signal-prediction evaluation used a 12-hour window and 94-dimensional representations.Source: §4 · p. 3

Reconstructing past and predicting future

  • Forecaster representations perform better on future prediction, and attention further improves performance.Representations were evaluated by a 1-layer LSTM on reconstruction of the past 12 hours or prediction of the next 12 hours, using MSE.Source: §4 · pp. 3–4, Table 1 · p. 4

Predicting mortality and discharge status within the next 24 hours

  • The clinical tasks were 24-hour stable ICU discharge and mortality prediction, evaluated with AUROC and AUPRC using representation inputs and raw-signal baselines.The baselines included a 3-layer LSTM trained on the original signals.Source: Table 2 · p. 4, §4 · p. 4

Improved performance in limited data setting

  • With scarce labels, representation-based classifiers outperform end-to-end supervised models; even with all available labels, the best unsupervised representations outperform supervised representations.Performance does not saturate across the tested training-set sizes, although purely supervised models might eventually surpass learned representations given more data.Source: Figure 3 · p. 4, §4 · p. 4

5 Conclusion

The paper proposes an attention-equipped forecasting Seq2Seq model for unsupervised patient time-series representations. In scarce-label settings, these representations improve clinical outcome prediction over end-to-end supervised learning on raw signals.

5 Conclusion

  • S2S-F-A, a forecasting Seq2Seq architecture with attention, produces the strongest representations for future signal prediction and clinical outcome prediction among the baselines studied.In scenarios with scarce labeled medical time-series data, classifiers trained on unsupervised representations gain performance over end-to-end supervised learning on raw inputs.Source: §5 · p. 5

A.1 Data

The experiments use eICU patient time series, resampled hourly across a selected set of 94 variables.

A.1 Data

  • The study used the eICU Collaborative Research Database v1.2 and extracted 94 variables, including vital signs and irregularly measured laboratory tests.Variables were included when at least 10% of the cohort had at least one recorded value.Source: §A.1 · p. 7
  • The raw time series were resampled onto a 60-minute grid, with implausible values rejected and missing values imputed online.Imputation used recent values, patient-history means, or population medians, depending on available history and measurement type.Source: §A.1 · p. 7
  • The dataset comprised 20,878 patients with 72–240 hours of history.Source: §A.1 · p. 7

A.1.1 Cohort selection

Cohort selection restricted the analysis to patients with a single ICU stay lasting 3–10 days, yielding 20,878 patients or stays.

A.1.1 Cohort selection

  • The cohort included only patients with one ICU stay, excluding stays shorter than 3 days or longer than 10 days.These filters yielded 20,878 patients or ICU stays.Source: §A.1.1 · p. 7

A.1.2 Data splits

The data were split by patient into five random train/validation/test replicates, with each patient’s entire record assigned to one set. Representations and classifiers were trained, tuned, and evaluated on distinct sets.

A.1.2 Data splits

  • Five random patient-level partitions each used train, validation, test1, and test2 sets at a 40:40:10:10 ratio.Each patient’s entire record appeared in exactly one set.Source: §A.1.2 · p. 7
  • Representations were trained on the training set, with representation-method hyperparameters tuned on the validation set.Source: §A.1.2 · p. 7
  • Classifiers used validation-set representations for training, test1 representations for hyperparameter optimization, and unseen test2 representations for performance evaluation.Five independent experiments were performed on the partition replicates.Source: §A.1.2 · p. 7

A.2 Representation learning

The appendix describes representation training and evaluation, model complexity, and a sweep of representation dimensions. The dimension results indicate that low-dimensional S2S-F-A representations retain reasonable predictive performance relative to raw features.

A.2 Representation learning

  • Feature columns were standard-scaled before representation-model training, and deep models used validation-based early stopping and grid search for hyperparameters.Recurrent autoencoders used batches of four patients with full histories; basic autoencoders used batches of 512 records.Source: §A.2 · p. 7

A.3 Representation evaluation

  • The first 12 hours of each recording were excluded from future-signal and task-prediction evaluation to avoid model-specific handling of incomplete histories.Source: §A.3 · p. 8

A.4 Model complexity

  • Seq2Seq model parameter counts do not depend on the length of the input time series being compressed.Table A2 compares the unsupervised representation-learning models; its S2S hidden-state dimension m equals the representation dimension.Source: §A.4 · p. 8, Table A2 · p. 8

A.5 Impact of representation dimension

  • The dimension sweep varies m from the standard 94-dimensional representation to assess compression while retaining prediction performance.The standard representation has a compression factor of 12 for windows of 12 hourly measurements of 94 variables.Source: §A.5 · p. 8
  • Low-dimensional S2S-F-A representations still obtain reasonable AUROC performance compared with raw features.Table A3 reports LSTM-1 AUROC scores across representation dimensions.Source: §A.5 · p. 8, Table A3 · p. 8

Related papers

Loading 1812.00490v1…