Source-linked AI summary
Self-Supervised Transformer for Sparse and Irregularly Sampled Multivariate Clinical Time-Series
Sindhu Tipirneni, Chandan K. Reddy
TL;DR
Sparse, irregular clinical time-series make aggregation and imputation-based representation learning costly and potentially lossy, while labeled patient data are limited. STraTS instead models observations as time-variable-value triplets with continuous embeddings and Transformer attention, and pretrains through forecasting on unlabeled data. The paper reports better mortality-prediction performance than state-of-the-art baselines, particularly with limited labels, while I-STraTS identifies important measurements with a small performance trade-off.
Problem
Sparse and irregular clinical time-series and limited labeled data challenge representation learning for mortality prediction.
Method
STraTS represents series as observation triplets, embeds continuous times and values, uses Transformer attention, and pretrains with forecasting on unlabeled data.
Results
STraTS outperforms state-of-the-art baselines for mortality prediction, especially when labeled data are limited; I-STraTS identifies important measurements with a slight performance compromise.
Takeaways & Limitations
The framework provides a representation for sparse, sporadic clinical series and an interpretable variant for identifying important measurements.
Takeaways & Limitations
The evaluation covers binary classification tasks, and longer series may make attention matrices computationally infeasible.
Abstract
from arXiv · showhide
Multivariate time-series data are frequently observed in critical care settings and are typically characterized by sparsity (missing information) and irregular time intervals. Existing approaches for learning representations in this domain handle these challenges by either aggregation or imputation of values, which in-turn suppresses the fine-grained information and adds undesirable noise/overhead into the machine learning model. To tackle this problem, we propose a Self-supervised Transformer for Time-Series (STraTS) model which overcomes these pitfalls by treating time-series as a set of observation triplets instead of using the standard dense matrix representation. It employs a novel Continuous Value Embedding technique to encode continuous time and variable values without the need for discretization. It is composed of a Transformer component with multi-head attention layers which enable it to learn contextual triplet embeddings while avoiding the problems of recurrence and vanishing gradients that occur in recurrent architectures. In addition, to tackle the problem of limited availability of labeled data (which is typically observed in many healthcare applications), STraTS utilizes self-supervision by leveraging unlabeled data to learn better representations by using time-series forecasting as an auxiliary proxy task. Experiments on real-world multivariate clinical time-series benchmark datasets demonstrate that STraTS has better prediction performance than state-of-the-art methods for mortality prediction, especially when labeled data is limited. Finally, we also present an interpretable version of STraTS which can identify important measurements in the time-series data. Our data preprocessing and model implementation codes are available at https://github.com/sindhura97/STraTS.
1 INTRODUCTION
Clinical time-series are sparse, irregularly sampled, and often supported by limited labels, making standard aggregation and imputation problematic. STraTS addresses these challenges with observation-triplet representations, continuous embeddings, Transformer attention, and forecasting-based self-supervision.
- Challenges: Clinical measurements are sparse and sporadic because variables differ in whether and how frequently they are observed.These characteristics arise from patient-specific measurement needs and underlying clinical conditions.
- Challenges: Aggregation and imputation can suppress fine-grained information while adding noise and computational overhead, especially for increasingly sparse series.Their effectiveness also depends on suitable imputation or interpolation schemes.
- STraTS: STraTS represents each series as observation triplets containing time, variable, and value rather than as a dense feature-by-time matrix.This avoids aggregation and imputation of unavailable observations.
- STraTS: Continuous Value Embedding encodes continuous times and measured values without binning, preserving fine-grained temporal information.A Transformer with multi-head attention then learns contextual triplet embeddings.
- Self-supervision and interpretability: Forecasting serves as a self-supervised proxy task that uses unlabeled data to learn more generalized and robust representations.The model also includes I-STraTS, an interpretable variant that identifies important measurements while slightly compromising performance metrics.
- Evaluation: The paper reports better mortality-prediction performance than competitive baselines on two real-world clinical datasets.Its stated contributions include improved performance under limited labels and an interpretable model variant.
2 RELATED WORK
Prior clinical time-series methods use aggregation, interpolation, Gaussian processes, or modified recurrent architectures to address missingness and irregular timing. These approaches can lose information, add noise or computation, and still face the challenge of limited labeled data, motivating self-supervision.
- Clinical time-series methods: Aggregation and imputation-based classifiers can ignore informative missingness and lose fine-grained temporal information.Missingness indicators can improve phenotyping performance, but aggregation remains a limitation.
- Clinical time-series methods: Gaussian-process approaches model irregular series through smooth curves or uncertainty-aware kernels, typically separating representation learning from classification.Their pipelines optimize Gaussian-process parameters before training the classifier.
- Clinical time-series methods: Learnable interpolation followed by classification produces dense series that either lose information with few interpolation points or add noise and computation with many.The interpolation and classification networks are stacked end to end.
- Clinical time-series methods: Time-aware recurrent models adjust hidden states or inputs for irregular gaps and missing values, but sequence lengths can become large and slow computation.Examples include T-LSTM, ODE-RNN, GRU-D, and DATA-GRU.
- Set-based methods: Set-based methods avoid dense interpolation, while attention aggregation in SeFT uses the same queries for all observations.The paper positions its observation-based design alongside this line of work.
- Self-supervised learning: Limited labeled data can cause supervised models to overfit and become more noise-sensitive, motivating self-supervised proxy tasks built from unlabeled data.Forecasting had shown promise as an auxiliary task for dense univariate time-series classification, while broader time-series use remained limited.
3 PROPOSED APPROACH
The proposed approach is presented as the STraTS model and its component architecture, with the detailed model description organized around the problem formulation and Figure 3.
- Proposed approach: The paper introduces STraTS by defining the prediction problem and then describing the model components illustrated in Figure 3.This section frames the architecture before detailing its individual components.
3.1 Problem Definition
The target task uses labeled ICU stays represented by demographic data and irregular observation triplets to predict in-hospital mortality. A larger dataset supports forecasting self-supervision, masking variables unobserved in each prediction window.
- Series representation: An observation triplet is a time, feature or variable, and value, and each multivariate series is a set of such triplets.This representation accommodates irregularly timed observations directly.
- Target task: Each labeled sample represents one ICU stay with demographics, a multivariate clinical series, and a binary in-hospital-mortality label.The target task predicts the label from the demographic vector and time-series.
- Forecasting task: Forecasting self-supervision uses a larger dataset containing the original series, a forecast mask, and observed forecast values.The dataset can include both labeled and unlabeled series through multiple observation windows.
- Forecasting task: The forecast mask identifies variables observed in the prediction window so unobserved forecasts are excluded from the loss.Figure 2 contrasts the fixed observation and prediction windows for mortality and forecasting tasks.
3.2 Architecture of STraTS
STraTS represents sparse, irregular clinical time-series as observation triplets and processes them with continuous embeddings, Transformer context encoding, attention-based fusion, and target/self-supervised prediction heads.
- Input representation: STraTS represents each time-series as observation triplets containing time, variable, and value rather than a dense matrix.This avoids aggregation and imputation of unavailable observations.
- Input representation: Continuous Value Embedding uses learnable one-to-many feed-forward networks to embed continuous times and measured values without categorization.The learned embeddings are more flexible than fixed-frequency sinusoidal encodings.
- Contextual triplet encoding: A multi-block Transformer with multi-head attention converts initial triplet embeddings into contextual triplet embeddings by attending to multiple embedding projections in parallel.Each block combines multi-head attention with a feed-forward network, residual connections, dropout, and layer normalization.
- Fusion and prediction: The concatenated time-series and demographics embeddings pass through a dense layer for target prediction, while forecasting uses the same architecture with a different prediction layer.The target task uses cross-entropy loss; forecasting uses masked mean squared error to account for missing forecast values.
- Fusion and prediction: Fusion self-attention assigns normalized attention weights to contextual triplets and combines them into a time-series embedding.The resulting embedding is concatenated with a demographics embedding before prediction.
- Self-supervision: The model is pretrained on forecasting and then fine-tuned on the target task.This self-supervised training stage uses forecast availability masks when computing the forecasting loss.
3.3 Interpretability
I-STraTS modifies STraTS so its output can be expressed as a linear combination of components derived from individual features, supporting identification of important measurements.
- Interpretable architecture: I-STraTS changes the architecture so its output is a linear combination of components derived from individual features.The model is designed to expose feature-level contributions.
- Interpretable architecture: I-STraTS combines initial triplet embeddings using Fusion Self-attention weights and uses the raw demographics vector as the demographics embedding.These changes distinguish it from the standard STraTS architecture.
- Feature contributions: I-STraTS assigns contribution scores to demographic and time-series features from their learned weights and input values.The demographic contribution score is given as w_o[j]d[j].
4 EXPERIMENTS
Experiments evaluate STraTS on two real-world EHR datasets for mortality prediction, including scarce-label settings and an interpretable variant. Results compare it with baseline models, assess self-supervision, and illustrate contribution scores for individual measurements.
- Datasets: Experiments use MIMIC-III and PhysioNet Challenge 2012 ICU time-series datasets for in-hospital mortality prediction.MIMIC-III uses the first 24 hours of data, while PhysioNet-2012 uses the first 48 hours.
- Baselines and Setup: Baseline comparisons include GRU, TCN, GRU-D, InterpNet, and SeFT under standardized preprocessing and optimization settings.Baselines use aggregated or interpolated representations, while model training uses Adam and normalized measurements and demographics.
- Prediction Performance: 3.2% and 3.5% PR-AUC improvements over the best baseline are reported on MIMIC-III and PhysioNet-2012, respectively, using 50% labeled data.Each model is evaluated across 10 random samplings, while STraTS additionally uses unlabeled data for self-supervision.
- Generalization: STraTS retains a crucial advantage as labeled-data percentages decrease, although performance declines for all models.The authors attribute STraTS’s scarce-label advantage to self-supervision.
- Ablation Study: Adding interpretability slightly reduces prediction scores, while self-supervision improves both STraTS and I-STraTS and can compensate for that reduction.I-STraTS with self-supervision outperforms STraTS without self-supervision on all MIMIC-III metrics and on PhysioNet-2012 PR-AUC.
- Interpretability: In a MIMIC-III case study, I-STraTS assigns high mortality probability using first-day data and highlights abnormal Lactate, LDH, Platelet count, and RDW measurements.Contribution scores increase with higher Lactate and LDH, increased blood glucose, and progressively low GCS-verbal values.
5 CONCLUSION
STraTS addresses sparse and sporadic clinical time-series through observation triplets, continuous value embeddings, Transformer attention, and forecasting-based self-supervision. The paper reports stronger generalization with scarce labels and identifies longer sequences as a computational boundary for future work.
- STraTS represents clinical time-series as observation triplets and uses CVE with multi-head attention to learn contextual representations without aggregation or imputation.
- Forecasting-based self-supervision using unlabeled data enables STraTS to learn more generalized representations and outperform state-of-the-art baselines.
- STraTS generalizes well when labeled data is scarce and is more robust to noise than existing methods.
- I-STraTS provides interpretability, while self-supervision compensates for the prediction-performance drop introduced by interpretability.
- Longer time series remain a computational boundary because their attention matrices can become large and infeasible.