PaperlayerView exact revision on arXiv

Source-linked AI summary

Interpolation-Prediction Networks for Irregularly Sampled Time Series

Satya Narayan Shukla, Benjamin M. Marlin

arXiv:1909.07782v1cs.LGstat.ML

TL;DR

The paper tackles supervised learning from sparse, irregularly sampled multivariate time series, especially physiological EHR data. It introduces a semi-parametric interpolation network that shares information across dimensions and feeds a modular prediction network, producing multiple representations for standard deep models. Across classification and regression tasks, the framework achieves statistically significant improvements over baseline and state-of-the-art methods.

  • Problem

    Supervised learning from sparse and irregularly sampled multivariate time series is challenging, particularly for physiological EHR data with unaligned observations.

  • Method

    The framework combines a semi-parametric interpolation network with a modular prediction network and produces regularly spaced outputs representing trends, transients, and observation intensity.

  • Results

    Statistically significant improvements were achieved for both classification and regression tasks over a range of baseline and state-of-the-art methods.

  • Takeaways & Limitations

    The modular framework accommodates sparse, irregular inputs while allowing standard deep learning prediction networks to operate on regularly spaced, fully observed outputs.

Abstract

from arXiv · show

In this paper, we present a new deep learning architecture for addressing the problem of supervised learning with sparse and irregularly sampled multivariate time series. The architecture is based on the use of a semi-parametric interpolation network followed by the application of a prediction network. The interpolation network allows for information to be shared across multiple dimensions of a multivariate time series during the interpolation stage, while any standard deep learning model can be used for the prediction network. This work is motivated by the analysis of physiological time series data in electronic health records, which are sparse, irregularly sampled, and multivariate. We investigate the performance of this architecture on both classification and regression tasks, showing that our approach outperforms a range of baseline and recently proposed models.

1 INTRODUCTION

The paper introduces Interpolation-Prediction Networks for supervised learning from sparse, irregularly sampled multivariate time series, motivated especially by physiological EHR data. Its semi-parametric interpolation network shares information across dimensions, produces multiple timescale representations, and feeds a modular prediction network; experiments cover classification and regression.

  • 1 INTRODUCTION: Interpolation-Prediction Networks combine semi-parametric interpolation layers with a prediction network that can use any standard deep learning model.The experiments use GRU networks as the prediction network.
  • 1 INTRODUCTION: Information from each input time series contributes to interpolating all other time series through end-to-end learning with supervised and unsupervised objectives.The interpolation network serves a role similar to a multivariate Gaussian process without requiring a positive definite covariance matrix.
  • 1 INTRODUCTION: Three outputs represent each input series: smooth broad trends, short-timescale transients, and an intensity function for local observation frequencies.The architecture explicitly separates transient events from broader trends and models observation-time patterns through a semi-parametric intensity representation.
  • 1 INTRODUCTION: Physiological EHR time series are commonly sparse, irregularly sampled, and unaligned across variables, motivating the proposed architecture.Dense physiological data streams remain uncommon in hospital systems.
  • 1 INTRODUCTION: Across two datasets and classification and regression tasks, the approach outperforms simple baselines and several previously proposed models.The evaluation includes comparisons with GRU models, a Gaussian process adapter, and a multi-task Gaussian process RNN classifier, plus ablations of the information channels.

2 RELATED WORK

The related work addresses supervised learning with sparse, irregularly sampled multivariate time series, which occur in EHRs and other domains and differ from fixed-dimensional missing-data problems. Prior approaches include discretization, imputation, Gaussian-process interpolation, and recurrent models, whereas this paper uses modular semi-parametric interpolation layers that produce regular inputs for standard networks.

  • 2 RELATED WORK: Sparse, irregularly sampled multivariate time series contain observations separated by large, irregular intervals and arise in EHRs, climate science, ecology, biology, and astronomy.The problem is distinct from ordinary missing data because observations typically occur in continuous time without a normal sampling frequency.
  • 2 RELATED WORK: Discretizing time converts irregular sampling into missing data but requires choosing an interval length, creating trade-offs between empty intervals and multiple observations per interval.Multiple observations in one interval require ad-hoc handling, while shorter intervals can leave many intervals empty.
  • 2 RELATED WORK: Observation times can themselves be informative, so prior methods incorporate timestamps or observation indicators into supervised models.Examples include GRU variants that decay past values or hidden states over time and models that use observation-time information.
  • 2 RELATED WORK: Gaussian-process approaches provide global interpolation for continuous-time data but can be computationally expensive and difficult to design for multivariate covariance structures.Deep networks can be stacked on Gaussian-process layers, but the interpolation computations and covariance design remain challenging.
  • 2 RELATED WORK: The proposed architecture uses semi-parametric deterministic feed-forward interpolation layers that output regularly sampled interpolants for arbitrary, unmodified deep classification and regression networks.This preserves modularity compared with approaches that directly modify recurrent architectures.

3 MODEL FRAMEWORK

The model framework combines a semi-parametric interpolation network with a modular prediction network to transform sparse, irregular multivariate series into fixed-size representations. Its interpolation outputs share information across dimensions, separate trends from transients, encode observation intensity, and are learned with supervised and masked reconstruction objectives.

  • Architecture: The architecture uses an interpolation network followed by a prediction network that can be any standard supervised neural network.The interpolation network operates on sparse, irregular multivariate inputs, while the prediction network produces the target prediction.
  • Interpolation outputs: The interpolation network produces three outputs per input dimension: smooth trends, transients, and observation intensity.These outputs are represented on regularly spaced reference times and supplied to the prediction network.
  • Interpolation outputs: The first interpolation layer applies separate low-pass, high-pass, and intensity transformations using semi-parametric RBF networks.The smooth and non-smooth interpolations use squared exponential kernels with different bandwidth parameters.
  • Interpolation outputs: The second interpolation layer merges information across dimensions through learnable correlations to produce cross-dimension interpolants and transient components.The transient component is defined as the difference between the high-pass interpolation and the smooth cross-dimension interpolation.
  • Prediction network: The interpolation network re-represents every input as a fixed-sized (DC) × T array on regularly spaced reference times.This representation allows recurrent, convolutional, or feedforward prediction networks to process the interpolated data.
  • Learning: Training combines supervised prediction loss with an autoencoder-like interpolation loss and ℓ2 regularization for both networks.Observed points are masked during learning so reconstruction evaluates held-out values rather than allowing trivial memorization.

4 EXPERIMENTS AND RESULTS

The experiments evaluate the architecture on UWaveGesture and MIMIC-III classification and regression tasks, comparing it with conventional, GRU-based, and Gaussian-process baselines. The proposed model performs strongly across these evaluations, with competitive UWaveGesture accuracy and consistently best average MIMIC-III metrics.

  • Datasets and tasks: The study evaluates classification and regression on UWaveGesture and MIMIC-III, including mortality and length-of-stay prediction.UWaveGesture is used for univariate gesture classification, while MIMIC-III provides multivariate physiological signals for mortality classification and length-of-stay regression.
  • Baselines: The baselines include standard feature-based models, GRU variants, Gaussian-process models, and customized neural networks.Non-neural baselines include Logistic Regression, SVM, Random Forests, AdaBoost, Linear Regression, SVR, and Random Forest Regression.
  • UWaveGesture results: The proposed model achieves similar UWaveGesture performance to the Gaussian process adapter while training 50x faster.It also achieves much better accuracy than other GRU-based models with training time in approximately the same order.
  • MIMIC-III results: On MIMIC-III, the proposed model consistently achieves the best average score across all reported classification and regression metrics.Paired t-tests indicate statistically significant improvements over all baselines for every metric except median absolute error, with p < 0.01.
  • Ablation results: Using only the transients and intensity outputs further improves regression performance with statistically significant gains over all baselines.The full model used for the main experiment includes smooth interpolation, transients, and intensity-function outputs.

5 DISCUSSION AND CONCLUSIONS

The paper presents a modular framework that interpolates sparse, irregularly sampled inputs before applying a prediction network. Its results show statistically significant improvements over baseline and state-of-the-art methods on both classification and regression tasks.

  • Framework: The framework separates interpolation from prediction, producing regularly spaced, fully observed multichannel inputs for an otherwise standard prediction network.This modular design addresses sparse and irregular inputs while allowing the prediction component to operate on the interpolated representation.
  • Contributions: The approach introduces semi-parametric feed-forward interpolation and decomposes irregular inputs into multiple distinct information channels.These channels support the framework's representation of different aspects of the input time series.
  • Conclusions: The results show statistically significant improvements over a range of baseline and state-of-the-art methods for both classification and regression.This conclusion summarizes the reported performance across the evaluated tasks.

A.1.1 MIMIC-III DATASET

MIMIC-III provides sparse, irregularly sampled physiological data for mortality classification and length-of-stay regression using the first 48 hours of each admission.

  • Dataset: The experiments use 12 physiological variables from 53,211 MIMIC-III admission records after excluding stays shorter than 48 hours.The dataset was collected from approximately 58,000 admissions at Beth Israel Deaconess Medical Center between 2001 and 2012.
  • Representation: Each admission is represented as a sparse, irregularly sampled 12-dimensional time series aligned over the first 48 hours.The time representation uses the union of timestamps observed across input dimensions.
  • Classification: Mortality classification labels indicate whether a patient died or was discharged after the first 48 hours.There are 4,310 positive mortality labels, representing 8.1% of patients.
  • Regression: Length-of-stay regression predicts the logarithm of stay duration in days and converts predictions back to days for reporting.The log transformation accommodates very long stay durations.

A.1.2 UWAVE DATASET

UWave is an univariate gesture-recognition dataset sparsified by retaining 10% of observations from each time series.

  • Dataset: The UWave dataset contains simple gesture patterns divided into eight categories.It has 3,582 training instances and 896 test instances, with 30% of training data used for validation.
  • Dataset: Each time series contains 945 observations before sparsification.The preparation follows Li & Marlin (2016).
  • Preparation: Randomly sampling 10% of each series’ observation points creates the sparse and irregularly sampled inputs.The resulting setup supports evaluation of training time and test accuracy against baseline models.

A.2.1 PROPOSED MODEL

The proposed framework combines interpolation, prediction, and observation-pattern information, then evaluates the resulting representations through training procedures and ablations across benchmark tasks.

  • Training: Artificial missing-data masks are resampled during mini-batch training, with 20% of observed points held out from every input time series.This supports the interpolation loss during learning.
  • Interpolation: For entirely missing time series, the first interpolation layer starts from the global mean while the second layer uses learned correlations from other channels.This preserves cross-channel interpolation when a channel has no observations.
  • Training: The framework trains interpolation and prediction components with a composite objective combining supervised prediction loss, interpolation loss, and parameter regularization.Prediction uses cross-entropy for classification or squared error for regression, while interpolation uses squared error.
  • Ablation analysis: Ablations evaluate all subsets of smooth interpolation, transients, and intensity outputs for mortality classification and length-of-stay regression.The full three-output representation is the reference configuration for the ablation analysis.
  • Ablation results: Smooth interpolation performs best individually for classification, whereas intensity is the strongest single source for regression and slightly exceeds the listed baselines in mean performance.Transient output alone performs significantly worse than smooth interpolation or intensity for classification.
  • Ablation results: Using all three outputs gives the best classification performance, while intensity improves median absolute error and intensity plus transients improves explained variance in regression.Combining smooth interpolation and intensity significantly boosts classification relative to either alone; smooth interpolation carries little regression information.
  • Benchmark evaluation: The benchmark comparison evaluates in-hospital mortality classification against multiple baselines on a dataset of adult patients.The benchmark uses first-48-hour data and compares models after sparse time series are discretized into one-hour intervals for baselines.
Loading 1909.07782v1…