PaperlayerView exact revision on arXiv

Source-linked AI summary

GRU-ODE-Bayes: Continuous modeling of sporadically-observed time series

Edward De Brouwer, Jaak Simm, Adam Arany, Yves Moreau

arXiv:1905.12374v2cs.LGstat.ML

TL;DR

Sporadically observed multidimensional time series are difficult to model because measurements are irregular across time and variables. The paper combines continuous-time GRU dynamics with Bayesian updates for incoming observations. GRU-ODE-Bayes exactly represents relevant Fokker-Planck dynamics and outperforms other methods across synthetic and real-world evaluations, with continuity especially useful in small-sample settings.

  • Problem

    Multidimensional time series are often sampled irregularly across time and variables, challenging methods that assume systematic fixed-interval measurements.

  • Method

    GRU-ODE-Bayes combines a continuous-time GRU with a Bayesian update network that processes sporadic observations within continuous ODE dynamics.

  • Results

    GRU-ODE-Bayes exactly represents Fokker-Planck dynamics for Ornstein-Uhlenbeck processes and outperforms the state of the art on healthcare and climate data.

  • Takeaways & Limitations

    The continuity prior is particularly important in the small-sample regime relevant to modestly sized clinical datasets.

  • Takeaways & Limitations

    The work focuses on Gaussian observations, although the method can be extended to binomial and multinomial observations.

Abstract

from arXiv · show

Modeling real-world multidimensional time series can be particularly challenging when these are sporadically observed (i.e., sampling is irregular both in time and across dimensions)-such as in the case of clinical patient data. To address these challenges, we propose (1) a continuous-time version of the Gated Recurrent Unit, building upon the recent Neural Ordinary Differential Equations (Chen et al., 2018), and (2) a Bayesian update network that processes the sporadic observations. We bring these two ideas together in our GRU-ODE-Bayes method. We then demonstrate that the proposed method encodes a continuity prior for the latent process and that it can exactly represent the Fokker-Planck dynamics of complex processes driven by a multidimensional stochastic differential equation. Additionally, empirical evaluation shows that our method outperforms the state of the art on both synthetic data and real-world data with applications in healthcare and climate forecast. What is more, the continuity prior is shown to be well suited for low number of samples settings.

1 Introduction

Real-world multidimensional time series are often sampled irregularly across time and variables, challenging methods built for fixed, systematic measurements. GRU-ODE-Bayes addresses this with continuous ODE dynamics tightly integrated with Bayesian processing of sporadic observations.

  • Motivation: Sporadic time series have irregular sampling times and missing variables at measurement events, violating assumptions of traditional recurrent methods.Clinical measurements illustrate this pattern because visits may be skipped and not every variable is recorded at each visit.
  • Motivation: Neural ODEs offer a continuous representation that can integrate dynamics over arbitrary time intervals, but prior uses largely generated observations through decoders or flows.The paper identifies a gap between continuous latent dynamics and direct processing of incoming sporadic observations.
  • Approach: GRU-ODE-Bayes interleaves continuous-time GRU dynamics with a Bayesian update network so incoming sporadic observations directly drive the latent dynamics.This tight integration replaces an encoder-decoder design in which ODE dynamics are decoupled from input processing.
  • Results: Tight coupling enables fine-grained nonlinear interactions, including inferring an unobserved variable’s jump from a correlated observed variable.The method is reported to quickly infer underlying stochastic-process parameters and variable correlations, unlike NeuralODE-VAE’s more general process structure.
  • Theory and evaluation: GRU-ODE-Bayes can exactly represent Fokker-Planck dynamics for Ornstein-Uhlenbeck processes and generalized versions, while outperforming the state of the art on healthcare and climate data.The paper frames the analysis around observations generated by multidimensional stochastic differential equations.

2 Proposed method

GRU-ODE-Bayes alternates continuous hidden-state evolution with Bayesian-style updates when sporadic observations arrive. Its design supports filtering, continuity, bounded hidden states, and end-to-end training from sporadically sampled data.

  • GRU-ODE-Bayes: The system combines GRU-ODE propagation between observations with GRU-Bayes updates when new observations become available.This creates an ODE with jumps at observation times.
  • GRU-ODE: GRU-ODE is derived from the GRU difference equation by taking a continuous-time limit for the hidden state.The resulting system is named GRU-ODE and has a minimal-GRU variant.
  • GRU-ODE: GRU-ODE supports continuous inputs when available and becomes autonomous when no continuous input signal is provided.Clinical trial drug doses are given as an example of a continuous input.
  • General properties of GRU-ODE: The hidden state remains within [-1, 1], with negative feedback stabilizing the system and correcting states that start outside this range.Boundedness makes the continuous dynamics compatible with GRU-Bayes and robust to numerical errors.
  • General properties of GRU-ODE: GRU-ODE is Lipschitz continuous with constant K = 2, encoding a continuity prior for the latent process.The paper reports that this prior is empirically important in the small-sample regime.
  • GRU-Bayes: GRU-Bayes preprocesses masked observations and updates the hidden representation, allowing adaptation within the GRU-ODE state region from a single observation.The update maps h(t−) to h(t+).
  • Objective function: The model trains with pre-jump negative log-likelihood and post-jump KL-divergence losses computed on observed dimensions.The post-jump objective encourages the model to mimic a Bayesian update, with the two losses combined using λ.
  • Implementation: For minibatches, hidden states are propagated jointly while updates and losses run only for series observed at each time; complexity is linear in observations and quadratic in observation dimension.The adjoint method can reduce memory use by avoiding backpropagation through solver operations.

3 Related research

Prior work commonly handles sporadic time series through binning, imputation, probabilistic models, or continuous decoders. These approaches leave limitations in processing irregular inputs, forecasting, parameter identification, or continuous-time modeling.

  • Irregular observations: Fixed-duration binning creates missing observations across time and features, so recurrent architectures typically require imputation, masks, and observation times.The resulting representation makes direct neural-network usage difficult.
  • Probabilistic models: Gaussian processes support imputation, latent-process modeling, representations, and forecasting, but usually have high uncertainty outside the observation range.The passage lists multitask Gaussian processes as the most popular probabilistic approach to missing data.
  • Neural ODE approaches: Neural ODE–VAE methods naturally generate irregularly sampled data but transfer sporadic-data processing to a discrete-time recurrent encoder.A concomitant VAE architecture uses a Neural ODE, while the cited earlier approach couples Neural ODEs with a VAE.
  • Sequential latent models: Discrete-time sequential-latent autoencoders use classical recurrent inference networks, making them less suited to sporadic data and forecasting.They have been useful for smoothing and counterfactual inference.
  • State-space methods: Extended Kalman filters model continuous-time distribution dynamics, but linearized state updates and parameter-identification difficulties limit practical applicability.The related discussion also notes that GRUs can learn long-term dependencies.
  • PDE-motivated architectures: PDE-motivated deep architectures generally result in discrete designs and do not explore continuous inputs and time.This distinguishes their motivation from continuous-time modeling of sporadic inputs.

4 Application to synthetic SDEs

On synthetic stochastic systems, GRU-ODE-Bayes captures interactions between variables and refines predictions as observations arrive. The paper also shows exact representation for multivariate Ornstein–Uhlenbeck dynamics and demonstrates handling of a nonlinear Brusselator-inspired system.

  • Synthetic OU process: Compared with NeuralODE-VAE, GRU-ODE-Bayes detects correlations between features and updates predictions more finely as new observations arrive.NeuralODE-VAE retrieves average sample dynamics, whereas GRU-ODE-Bayes uses incoming observations to refine predictions and confidence.
  • Synthetic OU process: An observation of Dimension 2 updates the prediction for Dimension 1 at time t = 3.This illustrates cross-feature updating in the multivariate process.
  • Filtering behavior: GRU-ODE-Bayes sequentially filters the hidden state and estimates the probability density of future observations, supporting long-term predictions.Sporadic inputs are fed directly into the ODE during this filtering process.
  • Theoretical and nonlinear dynamics: The model exactly represents the dynamics of multivariate Ornstein–Uhlenbeck processes with random variables and also handles nonlinear SDEs in a Brusselator-inspired example.The Brusselator example is described as a chaotic ODE.

5 Empirical evaluation

The model was evaluated for forecasting sporadic time series in healthcare and climate settings, using state-of-the-art baselines and cross-validation. GRU-ODE-Bayes outperformed the other methods on both datasets, with its continuity prior especially valuable when training samples were scarce.

  • Healthcare data: The healthcare task used 21,250 MIMIC-III patients and forecast the next three measurements after a 36-hour observation window.The cohort provided 96 longitudinal real-valued measurements over 48 hours after admission.
  • Climate data: The climate task used sporadic observations from 1,114 USHCN stations across five climate variables over a four-year window.The source data covered daily measurements collected over 150 years from stations across the United States.
  • Results: GRU-ODE-Bayes unequivocally outperformed all other methods on both the healthcare and climate datasets.Performance was assessed using negative log-likelihood and mean squared error with 5-fold cross-validation.
  • Impact of continuity prior: The continuity prior is particularly important when only a small number of samples is available.The paper frames this setting as relevant to rare-disease prediction, where data may be available for few patients.
  • Impact of continuity prior: In the small-sample MIMIC-III regime, GRU-ODE achieved higher accuracy than the discretized GRU version.The discretized model matched the continuous model on the full dataset, whereas the continuous cell performed better with fewer training patients.

6 Conclusion and future work

The paper combines continuous-time hidden-state evolution with Bayesian updates for sporadic observations and establishes continuity and representation properties. It reports strong performance on synthetic and real-world data while noting Gaussian observations as the primary scope.

  • Conclusion: GRU-ODE-Bayes combines GRU-ODE and GRU-Bayes to feed sporadic observations into continuous ODE dynamics for the data distribution.The system evolves the hidden state between observations and updates it when observations arrive.
  • Theoretical properties: The model can exactly represent Fokker-Planck dynamics for Ornstein-Uhlenbeck processes and generalized versions driven by stochastic differential equations.This is presented as a theoretical representation capability of the proposed filtering approach.
  • Future relevance: The continuity prior is particularly important in the small-sample regime relevant to modest-sized clinical datasets.The discretized model performed well on the full MIMIC-III dataset, but the continuous formulation was more valuable with fewer samples.
  • Scope: The experiments focused on time-series data with Gaussian observations, although the method can be extended to binomial and multinomial observations.The paper states that the corresponding negative log-likelihood and KL-divergence are analytically tractable for those extensions.
  • Continuity: GRU-ODE is Lipschitz continuous with constant K = 2, encoding a continuity prior for the latent process.The bound follows from bounded hidden states and bounded sigmoid and hyperbolic-tangent functions in the ODE.

C Comparison of numerical integration methods

The implementation compares Euler, explicit midpoint, and Dormand-Prince integration for the continuous GRU-ODE. Adaptive integration reduces the number of steps, while Euler remains computationally competitive and DOPRI offers greater numerical stability.

  • Integration methods: DOPRI is an adaptive step-size method using fourth- and fifth-order Runge-Kutta solvers.Adaptive stepping tunes the number of integration steps to reach the desired time point.
  • Step allocation: Adaptive integration used half as many time steps as the compared fixed-step approach on the same data and ODE.More steps occur near observations, while smoother dynamics permit larger steps.
  • Computational trade-off: Euler was more than competitive because each step required significantly fewer computations than DOPRI.This observation held on the data and simulations considered in the paper.
  • Computational trade-off: DOPRI may be preferred as the default method because of its better numerical stability.The paper presents this as a trade-off against Euler’s lower per-step computational cost.
  • Observation processing: The GRU-Bayes preprocessing maps the hidden state to observation-distribution parameters before incorporating sporadic inputs.For Gaussian observations, the parameters include means and log-variances, and the update can process dimensions jointly or sequentially.

H Ablation study of GRU-Bayes

The study compares the GRU-Bayes module with an MLP replacement and evaluates GRU-ODE-Bayes on correlated Ornstein–Uhlenbeck processes under varying parameters, lags, and correlations.

  • Ablation study of GRU-Bayes: Replacing GRU-Bayes with a two-layer MLP produces worse performance on the MIMIC forecasting task.The comparison uses a tanh hidden activation and linear output activation.
  • Representation capabilities: GRU-ODE-Bayes exactly represents the Fokker–Planck dynamics of the OU process when its distribution is Gaussian with time-dependent mean and covariance.The analysis is conditioned on a previous observation and expresses the distributional evolution through mean and variance dynamics.
  • Representation capabilities: GRU-ODE-Bayes models generalized Ornstein–Uhlenbeck processes by evolving the hidden state between observations and updating it when sporadic observations arrive.GRU-ODE stores continuous-time dynamics, while GRU-Bayes updates estimates such as sample-dependent parameters.
  • Experimental setup: The OU evaluation varies target parameters across samples, adds random time lags, and compares highly correlated dimensions with independent dimensions.The correlated setup uses ρ = 0.99, while the independent setup uses ρ = 0.
  • Experimental setup: The training set contains 10,000 samples with an average of 20 observations scattered over 10 seconds, and forecasting is evaluated after t = 4.Methods are trained with negative log-likelihood, with MSE also reported, and compared against NeuralODE-VAE variants.

I.2.2 Empirical evaluation

GRU-ODE-Bayes updates predictions sequentially as sporadic observations arrive, using correlations between dimensions to refine unobserved variables and confidence estimates.

  • Empirical evaluation: GRU-ODE-Bayes updates a feature’s prediction and confidence even when only the correlated other feature is observed.For example, Dimension 1 is updated when Dimension 2 is observed at t = 3.
  • Empirical evaluation: Sequentially filtering the hidden state lets GRU-ODE-Bayes estimate the future-observation PDF and perform long-term predictions.This differs from NeuralODE-VAE’s strategy of storing the whole dynamics in a single vector before mapping it to the time series.
  • Empirical evaluation: GRU-ODE-Bayes performs better than the compared methods on all tested setups for both NegLL and MSE.Random time lags cause only marginal degradation in NegLL and MSE.
  • Empirical evaluation: When the dimensions are uncorrelated (ρ = 0), observing one dimension provides no information about the other and performance is lower.The result follows the absence of shared information between the dimensions.
  • Empirical evaluation: As more observations are processed, confidence intervals narrow and predictions converge toward the underlying process’s asymptotic distribution.The initial predictions have broad intervals and progressively refine the variance estimate.

J Application to synthetic nonlinear SDE: the Brusselator

The paper evaluates GRU-ODE-Bayes on a nonlinear stochastic Brusselator process and shows that its filtered predictions use observations to share information across dimensions.

  • Application to synthetic nonlinear SDE: the Brusselator: The stochastic Brusselator is a two-dimensional nonlinear SDE derived from the Brusselator ODE, with x and y as process dimensions and a and b as parameters.The underlying ODE becomes unstable when b > 1 + a.
  • Application to synthetic nonlinear SDE: the Brusselator: The simulated process uses correlated Brownian motions and unstable parameters a = 0.3 and b = 1.4.The training data consists of random samples from trajectories of length 50.
  • Application to synthetic nonlinear SDE: the Brusselator: Training observations are sampled sporadically at an average rate of 4 samples every 10 seconds.The dataset is formed from random samples of the simulated trajectories.
  • Application to synthetic nonlinear SDE: the Brusselator: For newly generated Brusselator samples, the model displays filtered means and standard deviations using only observations available before each prediction time.The predictions share information between both process dimensions by exploiting their correlation.

L.1 Cleaning and subsampling

The climate-data preprocessing removes poor-quality records, restricts the time period and centers, subsamples observations, and retains the final four years for analysis.

  • Cleaning and subsampling: After cleaning and subsampling, the USHCN dataset contains 1,114 centers and 386,068 unique observations.The observations are sporadically distributed over the final four years, averaging 346 observations per center.

M Small-sample regime: additional results

Additional MIMIC-III results show that the continuity prior strongly helps GRU-ODE-Bayes in the low-sample regime, with performance reported using negative log-likelihood. The evaluation used held-out testing, five-fold cross-validation, and shared folds across models.

  • Small-sample results: Negative log-likelihood results on MIMIC-III further show that the continuity prior strongly helps in the low-number-of-samples setting.Table 6 reports vitals forecasting results for this regime.
  • Case-study evaluation: The MIMIC case study reports results for retained longitudinal intensive-care features and vitals forecasting.
  • Experimental setup: Models used the same hidden-state dimension, with hyperparameters tuned on a 20% validation split for each fold.The tuned parameters included dropout, weight decay, and learning rate.
  • Experimental setup: Performance was assessed on a held-out test set using five-fold cross-validation, with average and standard-deviation results reported across folds.The same folds were reused for each compared model to support reproducible and fair comparisons.
Loading 1905.12374v2…