Source-linked AI summary
Set Functions for Time Series
Max Horn, Michael Moor, Christian Bock, Bastian Rieck, Karsten Borgwardt
TL;DR
Irregularly sampled and unaligned time series are difficult for methods built around regular sampling, synchronization, or imputation, especially in healthcare. SeFT reframes classification as learning over sets of observations, enabling parallel processing and direct handling of these data. It achieves competitive classification performance while substantially reducing runtime, though comparisons involving methods that can leak future information require caution.
Problem
Irregularly sampled and unaligned measurements challenge architectures that assume regular sampling, synchronization, or imputation, while missingness patterns may contain useful information.
Method
SeFT reframes each irregular time series as an unordered set of observations and applies differentiable set-function learning with independently computed element embeddings.
Results
SeFT delivers competitive classification performance with low runtime compared with many competitors across real-world healthcare datasets.
Takeaways & Limitations
Set-based classification is a promising approach for irregularly sampled data, offering scalable processing and per-observation contribution estimates.
Takeaways & Limitations
IP-NETS can leak future information through preprocessing, and a leakage-free variant was beyond the scope of the work.
Abstract
from arXiv · showhide
Despite the eminent successes of deep neural networks, many architectures are often hard to transfer to irregularly-sampled and asynchronous time series that commonly occur in real-world datasets, especially in healthcare applications. This paper proposes a novel approach for classifying irregularly-sampled time series with unaligned measurements, focusing on high scalability and data efficiency. Our method SeFT (Set Functions for Time Series) is based on recent advances in differentiable set function learning, extremely parallelizable with a beneficial memory footprint, thus scaling well to large datasets of long time series and online monitoring scenarios. Furthermore, our approach permits quantifying per-observation contributions to the classification outcome. We extensively compare our method with existing algorithms on multiple healthcare time series datasets and demonstrate that it performs competitively whilst significantly reducing runtime.
1. Introduction
Irregularly sampled and unaligned measurements are common in real-world settings such as healthcare, yet many existing methods assume regular sampling, synchronized modalities, or imputation. These constraints can discard informative missingness patterns or impose substantial computational costs.
- Problem: Irregular sampling, misalignment, and observation frequency can characterize real-world measurements and may themselves carry predictive information.The passages note that imputing missing values is not always desirable because presence, absence, and frequency may be informative.
- Problem: Many variable-length methods still assume regular sampling or aligned measurements across modalities, limiting their use in these settings.
- Limitations of Existing Methods: Imputation-based classification requires modeling system dynamics and can lose missingness patterns that may be crucial for prediction.
- Limitations of Existing Methods: Jointly optimizing imputation and classification adds computational overhead, causing poor scalability or high memory requirements.
- Proposed Approach: SEFT classifies irregularly sampled time series as sets of observations, directly accommodating irregular sampling and unsynchronized measurements.The approach relaxes the requirement that inputs be processed in sequence and extends differentiable set-function learning to classification.
2. Related Work
Existing approaches to irregular time series classification include discretizing irregular sampling as missing data, modeling continuous-time processes with kernels or Gaussian processes, and end-to-end imputation-classifier systems. The related work also identifies scalability and standardization barriers for some process-based methods.
- Irregular Sampling as Missing Data: Irregular sampling can be converted into a missing-data problem by discretizing the time axis into non-overlapping intervals.
- Frameworks Supporting Irregular Sampling: Kernel methods and Gaussian processes support irregular sampling by defining comparisons or models directly for continuous-time observations.Examples include kernels on irregularly sampled series and Gaussian-process models using mean and covariance functions.
- Process-Based Approaches: Hawkes-process approaches have shown strong performance in modeling and forecasting but lack a standardized general approach for time series classification.
- Process-Based Approaches: The authors could not perform a fair Hawkes-process comparison because the approach did not scale to their datasets.
- End-to-End Learning of Imputation Schemes: End-to-end imputation methods jointly train an imputation scheme and classifier with gradient-based discriminative learning.The GP Adapters framework trains Gaussian-process kernel parameters alongside a classifier and produces fixed-size representations.
3. Proposed Method
SEFT represents irregularly sampled, potentially non-synchronized time series as unordered sets of timestamped observations and classifies them with learned set functions and attention. Its independently computed observation embeddings support variable-length inputs and linear scaling in the number of observations.
- Notation & Requirements: Each time series is an unordered set of observations, with each observation represented by time, value, and modality indicator.The formulation permits differing numbers of observations, times, and modalities across time series.
- Notation & Requirements: SEFT targets classification when modalities may be missing at time points, rather than requiring synchronized measurements or imputed regular grids.A non-synchronized series can have fewer than D modalities observed at a given time.
- Our Model: The model learns an order-invariant set function by independently embedding observations, aggregating them, and mapping the resulting representation to class logits.The observation time remains part of each set element, so the input need not be processed in sequence order.
- Our Model: A time encoding maps each observation’s time to multiple trigonometric features before the observation is embedded.The encoding uses varying frequencies and treats the number of encoding steps and maximum timescale as hyperparameters.
- Attention-based Aggregation: Attention conditions observation weights on both a learned summary of the full set and the individual observation, using multiple heads to capture different aspects.Head-specific queries allow attention to focus on aspects such as the distribution of one or more modalities; head outputs are concatenated for classification.
- Attention-based Aggregation: Independent observation embeddings give SEFT runtime and memory complexity O(n), compared with O(n^2) for Transformer-style cross-element embedding.Equation 2 also yields runtime O(M) for a time series with M observations, because each embedding can be computed independently.
4. Experiments
Experiments compare SEFT-ATTN with six competing approaches across mortality and online sepsis prediction tasks. SEFT-ATTN remains competitive while offering favorable runtime trade-offs, observation-level interpretability, and robustness to future-information masking.
- Experimental setup: Experiments evaluate three healthcare datasets covering two mortality tasks and online sepsis-onset prediction.The mortality tasks involve imbalanced data, while P-Sepsis evaluates prediction of onset within the next 6–12 hours.
- Experimental setup: Six baselines—GRU-SIMPLE, GRU-DECAY, PHASED-LSTM, IP-NETS, TRANSFORMER, and LATENT-ODE—are compared using a unified training framework except for LATENT-ODE.LATENT-ODE received limited hyperparameter tuning because its runtime was considerably higher, potentially underestimating its performance.
- Mortality results: SEFT-ATTN consistently ranks among the first four methods by AUPRC, outperforming TRANSFORMER on M3-Mortality while using lower runtime.On P-Mortality, SEFT-ATTN is comparable to GRU-D and TRANSFORMER with comparable or lower runtime; IP-NETS has slightly higher AUROC but nearly three-fold higher runtime.
- Interpretability: SEFT-ATTN quantifies individual observation contributions, attending to abrupt invasive blood-pressure changes and dense regions of intermittent non-invasive measurements.These attention patterns were reviewed with a medical expert and reflect clinically relevant signals described in the paper.
- Online sepsis results: TRANSFORMER and IP-NETS achieve the highest unmasked P-Sepsis performance but decline when future information is masked, indicating reliance on leaked future information.SEFT-ATTN has the same performance in both evaluation scenarios because its architecture provides no means for current predictions to access future observations.
- Online sepsis results: GRU-SIMPLE achieves the highest performance on P-Sepsis, plausibly because the dataset is sampled relatively regularly in hourly bins.This regularity favors a model that is not specifically designed for irregular sampling.
5. Conclusion and Discussion
SEFT provides competitive classification performance on irregularly sampled and unaligned time series with low runtime, while attention improves interpretability. The authors identify distribution-shift and competitor-comparison limitations and propose more robust attention for large sets.
- Conclusion: SEFT achieves competitive performance with low runtime, although it does not outperform state-of-the-art models.The authors frame individual-observation modeling as a promising direction for irregularly sampled data.
- Conclusion: Combining individual-observation modeling with attention increases model interpretability for medical and healthcare applications.The attention mechanism permits quantifying the contribution of observations to classification outcomes.
- Discussion: 6.3% higher average AUPRC on validation data than test data suggests a validation–test distribution shift for M3-Mortality.The authors also note that not binning observations may make their task harder than the original setup.
- Discussion: LATENT-ODE performance may be underestimated because extensive hyperparameter search was not performed and its reconstruction–classification trade-off used transferred parameters.The authors state that dataset characteristics could make this trade-off suboptimal.
- Future work: Future work targets attention mechanisms for very high-cardinality sets and numerical robustness against catastrophic cancellation during aggregation.GPU implementations of Kahan summation are identified as one possible direction.
A. Appendix
The appendix reports label-prevalence tables for the two mortality tasks and the online sepsis prediction task.
- Dataset statistics: Table A.1 reports label prevalence for the binary M3-Mortality classification task.
- Dataset statistics: Table A.2 reports label prevalence for the binary P-Mortality classification task.
- Dataset statistics: Table A.3 reports label prevalence for the P-Sepsis online prediction task.
A.1. Dataset preprocessing
The preprocessing appendix describes dataset variables, splits, exclusions, and released tooling for reproducing the experiments.
- Filtering: Twelve P-Mortality patients lacking any time-series information were removed.
- Filtering: Three P-Sepsis files lacking static values or time-series information were excluded.
- Variables: The experiments used static variables such as age and gender, with categorical variables expanded to one-hot encodings.Table A.4 lists the static variables used for each dataset.
- Variables: All available time-series variables were used as continuous inputs without additional transformations.These included vitals, laboratory measurements, and interventions.
- Splits: Datasets were partitioned into training, validation, and testing subsets, using original MIMIC-III splits but new Physionet splits because challenge test sets were unavailable.For Physionet datasets, the full data were split into 80% training and 20% testing sets with stratification.
- Implementation: The authors released preprocessing and training packages to support direct comparison and reproducibility.The preprocessing package integrates with tensorflow-datasets.
A.2. Comparison partners
The comparison partners cover recurrent models for irregularity, interpolation-based models, and Transformer attention architectures.
- Recurrent models: GRU-SIMPLE augments each GRU input with a measurement mask and the time since the corresponding modality was last measured.
- Recurrent models: GRU-D modifies the GRU cell to decay past observations toward modality means using learnable decay rates and measurement masks.
- Recurrent models: PHASED-LSTM uses time-dependent gating for irregularly spaced event sequences but requires additional input augmentation for unaligned measurements.
- Interpolation models: IP-NETWORKS apply multiple semi-parametric interpolation schemes to create regularly sampled representations and train interpolation jointly with classification.The schemes capture long-term trends, transients, and sampling information.
- Attention models: The adapted Transformer encodes sequence elements simultaneously, captures interactions with multi-head attention, mean-aggregates final representations, and predicts class logits with a one-layer MLP.
A.3. Implementation details
Experiments used a common cached input-processing pipeline and fixed TensorFlow/GPU setup, except LATENT-ODE, whose separate evaluation introduced runtime and performance biases.
- All experiments used TensorFlow 1.15.2 and NVIDIA GeForce GTX 1080Ti GPUs.
- Model-specific representations and data transformations were cached to support fair comparisons between methods.
- LATENT-ODE used the authors’ implementation without cached input processing, PyTorch code, or a comparable hyperparameter search.Its runtime was an order of magnitude higher, making thorough search infeasible.
- These evaluation differences could bias runtime and performance comparisons involving LATENT-ODE.
A.4. Training, Model Architectures, and Hyperparameter Search
The study standardized optimization and tuned model-specific architectures across methods, while documenting special handling for static variables, Transformers, and Latent-ODE. Selected hyperparameters were reported for reproducibility.
- Training: All models used Adam, with learning rates sampled log-uniformly from 0.01 to 0.0001 and batch sizes from 32, 64, 128, 256, or 512.
- Recurrent neural networks: RNN-based methods varied hidden units and dropout, while PHASED-LSTM omitted dropout because learned frequencies were hypothesized to serve a similar function.
- Static variables: Static variables initialized RNN hidden states through a one-hidden-layer neural network matched to the RNN hidden size.
- SEFT-Attn: SEFT-Attn searched layers, layer widths, dropout, embedding width, and attention dropout across predefined ranges for its encoding and aggregation networks.
- Transformer: The Transformer concatenated each time-point vector with a measurement indicator and replaced missing modality values with zero.
- Latent-ODE: Latent-ODE reused author-selected PhysioNet 2012 hyperparameters across datasets because its training time was almost an order of magnitude longer.A thorough search was therefore not possible, and selected hyperparameters were listed in Table A.5 for reproducibility.