PaperlayerView exact revision on arXiv

Source-linked AI summary

Multi-Time Attention Networks for Irregularly Sampled Time Series

Satya Narayan Shukla, Benjamin M. Marlin

arXiv:2101.10318v2cs.LGcs.AI

TL;DR

Sparse, irregularly sampled multivariate time series challenge models built for fixed, fully observed inputs. The paper proposes Multi-Time Attention Networks, which learn continuous-time embeddings and attention-based fixed-length representations, and reports comparable or better interpolation and classification performance with much faster training.

  • Problem

    Sparse, irregular, and partially observed time series do not fit standard models that assume fully observed, fixed-size representations.

  • Method

    mTANs learn continuous-time embeddings and use time attention to interpolate observations into fixed-length representations for neural networks.

  • Results

    mTAN-based models perform as well or better than baseline and state-of-the-art methods on interpolation and classification, with training times one to two orders of magnitude faster.

  • Takeaways & Limitations

    The mTAN module provides a flexible interface between sparse, irregular time series and multiple deep neural-network architectures.

Abstract

from arXiv · show

Irregular sampling occurs in many time series modeling applications where it presents a significant challenge to standard deep learning models. This work is motivated by the analysis of physiological time series data in electronic health records, which are sparse, irregularly sampled, and multivariate. In this paper, we propose a new deep learning framework for this setting that we call Multi-Time Attention Networks. Multi-Time Attention Networks learn an embedding of continuous-time values and use an attention mechanism to produce a fixed-length representation of a time series containing a variable number of observations. We investigate the performance of this framework on interpolation and classification tasks using multiple datasets. Our results show that the proposed approach performs as well or better than a range of baseline and recently proposed models while offering significantly faster training times than current state-of-the-art methods.

1 INTRODUCTION

mTANs address sparse, multivariate, irregularly sampled time series by combining learned continuous-time embeddings with time attention. The framework re-represents variable-length observations at fixed reference points and supports interpolation and classification with competitive performance and faster training.

  • Irregular sampling challenges standard models because they assume fully observed, fixed-size representations, while observations may be sparse, unaligned, and separated by irregular intervals.
  • mTANs are continuous-time, interpolation-based models that learn time embeddings and use time attention instead of a fixed similarity kernel.This gives the model more representational flexibility than previous interpolation-based models.
  • The encoder re-represents irregular observations over reference points as a fixed-length latent representation, while the decoder reconstructs outputs from those representations.
  • mTANs target multivariate sparse series, including partially observed vectors, by learning temporal similarity from data rather than using fixed kernels.
  • The framework achieves interpolation and classification performance as good as or better than current methods while substantially reducing training time.

2 RELATED WORK

Prior approaches handle irregular time series through discretization, recurrent modifications, interpolation, latent continuous-time models, or attention. mTAN instead learns time-based similarity for interpolation and directly handles partially observed dimensions without separate imputation.

  • Irregular time series contain uneven observation intervals, possible cross-variable misalignment, and sparsity, creating difficulties for standard fixed-size supervised models.
  • Fixed temporal discretization is simple but requires ad-hoc handling of multiple observations per bin and creates missing data when bins are empty.
  • Recurrent approaches incorporate irregularity through time intervals, modified gates, or time gates, but represent a distinct family of solutions.
  • Interpolation-oriented methods use bidirectional recurrent models or semi-parametric RBF layers to exploit observations around target times.
  • Latent ODE approaches model observations through continuous-time latent functions, often using ODE-based encoders.
  • Attention-based alternatives encode time in self-attention, commonly replacing positional encoding with time representations.
  • mTAN learns similarity from time embeddings alone, interpolates observed values at query times, and attends only to observed dimensions without separate imputation.Unlike fixed-form kernels, learned time-attention similarity provides greater flexibility.

3 THE MULTI-TIME ATTENTION MODULE

The mTAN module embeds continuous time and uses attention-based interpolation to map sparse, irregular observations into fixed-dimensional representations. Its discretized outputs at reference points can interface with standard neural-network layers and encoder-decoder architectures.

  • The mTAN module re-represents sparse, irregular time series in a fixed-dimensional space using multiple continuous-time embeddings and attention-based interpolation.
  • Notation: For supervised data, each variable may have different observation times and counts, while interpolation tasks use the multivariate series without a target label.
  • Time Embedding: Continuous time points are embedded into multiple vector spaces, generalizing positional encoding from discrete positions to continuous time.
  • Time Embedding: Learnable periodic and linear time-embedding terms can represent periodicity and non-periodic patterns dependent on time progression.
  • Multi-Time Attention: The attention module maps a query time and sparse multivariate keys and values to a J-dimensional embedding.
  • Multi-Time Attention: Intermediate functions smooth each data dimension using weights from scaled dot-product attention between query and observed-time embeddings.
  • Multi-Time Attention: Learned time embeddings and multiple embedding dimensions make the temporal kernel and final representation flexible across input dimensions.
  • Discretization: Materializing the continuous output at reference time points produces a sequence of embeddings that can feed fully connected, recurrent, or convolutional layers.

4 ENCODER-DECODER FRAMEWORK

The mTAN network combines an encoder-decoder architecture with latent states defined at reference times to model irregularly sampled time series. Training uses variational objectives, optionally augmented with supervised classification.

  • Model Architecture: The model defines latent states at K reference time points, each initially distributed according to a standard multivariate normal prior.
  • Model Architecture: The three-stage decoder uses an RNN, mTAN interpolation at query times, and a fully connected decoder to parameterize the output distribution.The output distribution is a diagonal-covariance Gaussian with fixed variance σ^2.
  • Model Architecture: The encoder maps the input series through mTAN at reference points, applies an RNN for longer-range temporal structure, and constructs Gaussian latent distributions.The latent means and variances are produced by fully connected layers applied to the RNN outputs.
  • Learning: The unsupervised model maximizes a normalized variational lower bound on the log marginal likelihood for sparse, irregularly sampled series.The objective normalizes each data case by its total number of observations and accounts for dimensions unobserved at particular time points.
  • Learning: A supervised component uses latent states as features, with λ trading off supervised and unsupervised terms in the augmented objective.For classification, the paper uses a GRU followed by a two-layer fully connected network and approximates intractable expectations with a small number of samples.

5 EXPERIMENTS

The experiments evaluate mTAND-based models on interpolation and classification using three real-world datasets and comparisons with recurrent, interpolation, set-function, and ODE-based approaches. Results show strong predictive performance, faster training than ODE-based methods, and benefits from temporally distributed latent representations and learned time components.

  • Experimental setup: Experiments use PhysioNet Challenge 2012, MIMIC-III, and Human Activity datasets for interpolation and classification tasks.PhysioNet and MIMIC-III involve whole-time-series classification, while Human Activity classifies each time point.
  • Models and baselines: The evaluation compares mTAND-Full and mTAND-Enc with RNN-based, encoder-decoder, interpolation, set-function, and ODE-based models.Baselines include RNN-Impute, RNN-∆t, RNN-Decay, GRU-D, Phased-LSTM, IP-Nets, SeFT, and RNN-VAE.
  • Interpolation: mTAND-Full consistently and substantially outperforms previous approaches on PhysioNet interpolation across settings observing 50%–90% of values.The reported interpolation columns correspond to different conditioning settings and are not comparable with one another.
  • Classification: mTAND-Full and mTAND-Enc significantly improve PhysioNet mortality-prediction performance relative to ODE-RNN, L-ODE-ODE, and other baselines.The classification comparison is reported in Table 2.
  • Classification: mTAND-Full achieves better MIMIC-III mortality-prediction performance than IP-Net, SeFT, and RNN baselines, while ODE-based models have slightly higher but not statistically significant mean AUC.On PhysioNet classification, mTAND-Full is more than an order of magnitude faster than ODE-based methods.
  • Classification and representation analysis: mTAND-based classifiers significantly outperform baseline models on Human Activity, while temporally distributed latent representations improve synthetic-data interpolation mean squared error.The synthetic experiments report better capture of local structure than latent ODE-based methods encoding to a single time point.
  • Ablations: Learning time embeddings improves classification over fixed positional encodings, while learned similarity kernels perform as well as or better than fixed RBF kernels.These findings come from ablation experiments in Appendix A.1.

6 DISCUSSION AND CONCLUSIONS

The paper presents the mTAN module and a VAE-based encoder-decoder for sparse, irregularly sampled data. Across interpolation and classification, the resulting model performs as well as or better than baseline and state-of-the-art methods while training one to two orders of magnitude faster.

  • Conclusion: The paper presents the Multi-Time Attention module with a VAE-based encoder-decoder for sparse and irregularly sampled data.The mTAN module is described as an interface for such time series and multiple deep neural network architectures.
  • Conclusion: mTAN models perform as well as or better than baseline and state-of-the-art methods on interpolation and classification tasks.The conclusion summarizes results across both evaluated tasks.
  • Conclusion: Training times are one to two orders of magnitude faster than previous state-of-the-art methods.The conclusion identifies computational speed as a central result.
  • Future applicability: The mTAN module can interface sparse, irregular time series with architectures including GAN-based models, while convolutional alternatives may improve parallelism.These are proposed extensions beyond the VAE-based recurrent architecture studied here.

A.1 ABLATION STUDY

The ablations examine whether learned time embeddings and similarity kernels improve mTAN classification performance relative to fixed alternatives.

  • Time embedding: 1% AUC improvement results from learning the time embedding rather than using fixed positional encodings.The comparison uses mTAN-Full on PhysioNet and MIMIC-III classification tasks, averaging AUC over 5 runs.
  • Interpolation kernels: The interpolation-kernel ablation compares IP-Nets using fixed RBF kernels with mTAND-Enc using a learnable similarity kernel.The comparison covers classification on PhysioNet, MIMIC-III, and Human Activity, with average AUC reported over 5 runs.

A.2 SYNTHETIC INTERPOLATION EXPERIMENTS

The synthetic experiment tests interpolation from sparse, irregular observations, comparing mTAN with a Latent ODE model across trajectories and mean squared error.

  • Dataset construction: The synthetic dataset contains 1000 trajectories of 100 time points, with 10 fixed reference points used to construct local interpolations.Reference values are drawn from a standard normal distribution, and an RBF kernel with fixed bandwidth 100 constructs interpolations.
  • Evaluation setup: Twenty observations are sampled from each trajectory, with 80% used for training and 20% for testing.At test time, the encoder receives 20 irregularly sampled points and the decoder generates values at all 100 time points.
  • Interpolation results: mTAN captures local trajectory structure better than the Latent ODE model with ODE encoder, whose interpolations are much smoother.Figure 3 compares ground truth trajectories, mTAN reconstructions, and Latent ODE reconstructions for three examples.
  • Quantitative evaluation: Table 5 reports mean squared error for reconstruction and interpolation after conditioning on 20 irregularly sampled time points.The table compares mTAN with the best-performing Latent-ODE with ODE encoder baseline.

A.3 ARCHITECTURE DETAILS

The architecture combines multi-time attention with recurrent encoder-decoder components and task-specific output processing for interpolation and classification.

  • mTAN-Full: mTAN-Full uses bidirectional GRUs in both the encoder and decoder.The encoder maps recurrent hidden states at reference points to mean and variance, while the decoder independently decodes mTAN embeddings.
  • mTAND-Enc: mTAND-Enc places a Multi-Time attention module before a GRU and a two-layer fully connected classifier.The final hidden state is passed to the classifier to produce class probabilities.
  • Loss function: Training uses negative log-likelihood with fixed variance for reconstruction and cross-entropy loss for classification.The reconstruction variance is fixed at 0.01, and ELBO estimation uses 5 interpolation samples versus 1 classification sample.

A.4 HYPERPARAMETERS

The experiments use specified baseline and mTAN hyperparameter procedures, rescaled time inputs, publicly available datasets, and GPU-based execution.

  • Baselines: Baseline hyperparameters are reported for PhysioNet and Human Activity, while MIMIC-III baseline hyperparameters are independently tuned on validation data.The MIMIC-III search varies GRU hidden units, latent dimensions, fully connected units, and ODE-related settings.
  • mTAN settings: mTAN uses time embeddings of size 128 and searches the number of embeddings, latent dimension, GRU encoder size, and reference-point count.Classification uses 128 reference points; interpolation searches 8, 16, 32, 64, and 128 reference points.
  • Experiment materials: The displayed experiment materials include regularly sampled and irregularly sampled time points, attention-weight visualization, and dataset links.The cited materials also identify the publicly available PhysioNet, MIMIC-III, and Human Activity datasets.
  • Reproducibility: All experiments were run on an Nvidia Titan X GPU, and code for reproducing the results is publicly available.The implementation repository is identified in the cited passage.
Loading 2101.10318v2…