Source-linked AI summary
Attend and Diagnose: Clinical Time Series Analysis using Attention Models
Huan Song, Deepta Rajan, Jayaraman J. Thiagarajan, Andreas Spanias
TL;DR
Clinical prediction needs models that handle long clinical time series without the parallelization difficulty of recurrent computation. The paper introduces SAnD, an attention-only architecture with masked self-attention and temporal representations, plus a multi-task variant. On MIMIC-III benchmarks, SAnD achieves state-of-the-art performance across the evaluated tasks and generally outperforms LSTM-based RNNs.
Problem
Clinical time-series prediction involves long, irregular, and incomplete EHR sequences, while recurrent models are difficult to parallelize and benchmarks have lacked universal acceptance.
Method
SAnD models multivariate clinical time series solely with masked self-attention, positional encoding, dense interpolation, and a multi-task variant for joint inference.
Results
SAnD achieves state-of-the-art performance across MIMIC-III benchmark tasks and in most cases outperforms state-of-the-art LSTM-based RNNs.
Takeaways & Limitations
Attention mechanisms provide an effective and computationally efficient approach for clinical data modeling across a wide range of diagnosis tasks.
Abstract
from arXiv · showhide
With widespread adoption of electronic health records, there is an increased emphasis for predictive models that can effectively deal with clinical time-series data. Powered by Recurrent Neural Network (RNN) architectures with Long Short-Term Memory (LSTM) units, deep neural networks have achieved state-of-the-art results in several clinical prediction tasks. Despite the success of RNNs, its sequential nature prohibits parallelized computing, thus making it inefficient particularly when processing long sequences. Recently, architectures which are based solely on attention mechanisms have shown remarkable success in transduction tasks in NLP, while being computationally superior. In this paper, for the first time, we utilize attention models for clinical time-series modeling, thereby dispensing recurrence entirely. We develop the \textit{SAnD} (Simply Attend and Diagnose) architecture, which employs a masked, self-attention mechanism, and uses positional encoding and dense interpolation strategies for incorporating temporal order. Furthermore, we develop a multi-task variant of \textit{SAnD} to jointly infer models with multiple diagnosis tasks. Using the recent MIMIC-III benchmark datasets, we demonstrate that the proposed approach achieves state-of-the-art performance in all tasks, outperforming LSTM models and classical baselines with hand-engineered features.
Introduction
Clinical time-series prediction must handle long, irregular, and incomplete EHR sequences, while RNNs remain difficult to parallelize. SAnD addresses this gap with masked self-attention and temporal representations, evaluated across MIMIC-III benchmarks.
- Clinical data contain multivariate time-series observations with long-term dependencies, irregular sampling, and missing values.
- RNNs and LSTMs model clinical sequences effectively but their sequential computations make parallelization challenging.
- SAnD is an attention-only architecture using masked self-attention, positional encoding, and dense interpolation for clinical time-series modeling.
- The study evaluates SAnD on all four MIMIC-III clinical prediction benchmarks, which provide a public basis for rigorous comparison.
- SAnD achieves state-of-the-art prediction performance and is highly competitive with, or usually better than, LSTM-based RNNs.
- A multi-task SAnD variant jointly infers multiple diagnosis tasks and demonstrates effectiveness over RNNs in joint inferencing.
Related Work
Clinical sequence modeling has used LDS, Gaussian processes, and increasingly RNNs to address irregular, missing, heterogeneous, and long-range clinical data. SAnD extends attention-based sequence modeling from NLP to multivariate clinical time series.
- Clinical data modeling is challenged by irregular sampling, missing values, measurement errors, heterogeneous measurements, and long-range dependencies.
- Earlier clinical time-series approaches included Linear Dynamical Systems and Gaussian Processes for modeling challenging observations.
- RNN-based approaches used LSTMs and training strategies to address diagnosis tasks and missing values, while recurrent convolutional networks modeled EEG patterns.
- Attention mechanisms were commonly integrated with LSTM encoder-decoder systems in NLP before attention-only architectures matched RNN performance in sequence generation.
- SAnD proposes the first attention-based sequence-modeling architecture for multivariate time-series data and studies it in clinical diagnosis.
Proposed Approach
SAnD adapts attention-only sequence modeling to multivariate clinical measurements, motivated by memory and optimization considerations. It replaces recurrence with self-attention while retaining temporal information through architectural representations.
- SAnD is a fully attention-based approach that studies clinical prediction without recurrence, requiring modifications to Transformer components for multivariate time series.
- The architecture targets long clinical sequences by using attention to study the memory needed for benchmark medical-data tasks.
- Attention models are motivated by mathematical simplicity that can support additional constraints, including explicit correlations between measurements.
Architecture
SAnD models clinical time series without recurrence or convolutional sequence modeling, using masked self-attention, positional encoding, and dense interpolation to represent temporal structure efficiently.
- Architecture: SAnD replaces recurrent and convolutional sequence modeling with self-attention and dense interpolation, using stacked attention and feed-forward layers with residual connections.The attention module contains N identical layers.
- Architecture: Clinical measurements are embedded into vectors before sequence modeling, with positional encodings added to represent temporal order.The input embedding captures dependencies across variables, while positional encoding incorporates time-step order.
- Attention Module: Masked self-attention restricts each position to earlier time steps within a mask of size r, enabling causal dependency modeling.Queries, keys, and values are derived from position-encoded input embeddings.
- Attention Module: Multi-head attention creates multiple attention graphs that encode complementary dependency types.The architecture uses 8 attention heads.
- Dense Interpolation for Encoding Order: Dense interpolation produces a concise sequence representation of dimension d × M while preserving partial temporal order.When M = T, dense interpolation reduces to concatenation; weights determine each time step’s contribution to the final representation.
- Dense Interpolation for Encoding Order: Dense interpolation can be implemented as U = S × W, followed by stacking U’s columns to obtain the sequence representation.The interpolation weights are cached in W ∈ R^(T × M).
- Computational Properties: The proposed architecture requires O(1) sequential operations and O(T · r · d) computation per layer, compared with O(T) sequential operations for recurrent models.The attention computation is entirely parallelizable.
MIMIC-III Benchmarks & Formulation
The paper evaluates SAnD on four MIMIC-III clinical prediction tasks and specifies task formulations, dataset structure, model choices, and a joint multi-task variant.
- MIMIC-III Benchmarks & Formulation: MIMIC-III contains de-identified critical-care data from 2001–2012, organized into patient episodes with time-series events and episode-level outcomes.The benchmark cohort includes 33,798 unique patients and 42,276 hospital admissions and ICU stays.
- In Hospital Mortality: In-hospital mortality is binary classification over a 24-hour window, with labels indicating death during admission or discharge alive.The benchmark cohort’s mortality rate is 13%.
- Decompensation: Decompensation predicts whether a patient will die within the next 24 hours at every time step during continuous monitoring.The task uses fixed time windows and next-24-hour mortality labels.
- Length of Stay: Length-of-stay forecasting analyzes hourly events and converts remaining-stay values into ten classification buckets.Predictions are made once every hour from admission.
- Phenotyping: Acute-care phenotyping predicts 25 disease conditions as a multi-label classification problem because patients may have multiple diagnoses.The conditions include critical, chronic, and mixed categories.
- Applying SAnD to MIMIC-III Tasks: SAnD modeling requires choices for attention mask size r, interpolation factor M, and number of attention blocks N.Large N can increase memory requirements, although the paper reports state-of-the-art results with small N.
- Multi-task Learning: SAnD-Multi jointly evaluates losses for all four tasks while sharing input embeddings and attention modules across task-specific prediction layers.Different tasks may use different mask sizes and interpolation factors but must share the same N.
- Multi-task Learning: Figure 3 examines training behavior and how attention mask size, attention depth, and interpolation factor affect test performance.The figure covers the main SAnD hyperparameters evaluated on the MIMIC-III tasks.
Performance Evaluation
SAnD is evaluated on the MIMIC-III benchmark tasks against LSTM-based RNNs and logistic regression with hand-engineered features. Across the reported tasks, attention-mask and architecture choices affect performance, efficiency, and overfitting.
- Evaluation setup: The evaluation compares SAnD with state-of-the-art LSTM-based RNNs and logistic regression using hand-engineered features on MIMIC-III benchmark tasks.The study also examines single-task and multi-task strategies.
- Phenotyping: For phenotyping, performance improves as the attention mask grows, but shows no significant improvement beyond r = 96.The best reported settings were N = 2 and M = 120, and the model was highly competitive with state-of-the-art results.
- In Hospital Mortality: SAnD outperforms both baseline methods on in-hospital mortality, with best performance at N = 4 and M = 12.The task uses measurements from the last 24 hours and applies causal masking without additional masking.
- Decompensation: SAnD achieves significantly better results than the state-of-the-art on decompensation while using r = 24 for more efficient training than an LSTM model.The best results used N = 1 and M = 10, and increasing model capacity easily led to overfitting.
- Length of Stay: For length of stay, SAnD with r = 24 achieves superior results to LSTM across all evaluation metrics.The best settings were N = 3 and M = 12, with no further improvements at larger N.
- Multi-task evaluation: The multi-task SAnD variant produces the best performance in almost all cases and uses a lower phenotyping mask size of 72, improving training efficiency.The reported multi-task settings use N = 2, M = 36 for phenotyping, and M = 12 for the other tasks.
Conclusions
The paper presents SAnD as a recurrence-free architecture for clinical time-series modeling. It combines masked self-attention with positional encoding and dense interpolation to model temporal clinical data efficiently across diagnosis tasks.
- Architecture: SAnD models clinical time-series data solely with masked self-attention, dispensing with recurrence completely.Its self-attention captures dependencies within a sequence using a multi-head design.
- Temporal representation: Positional encoding and dense interpolation embeddings incorporate temporal order into the sequence representation.The approach is designed for clinical diagnosis tasks with varying data characteristics.
- Overall conclusion: SAnD achieves superior performance on challenging MIMIC-III benchmark datasets while maintaining an efficient training process.The paper presents this result across a wide range of clinical diagnosis tasks.