PaperlayerView exact revision on arXiv

Source-linked AI summary

GRU-ODE-Bayes: Continuous modeling of sporadically-observed time series

Edward De Brouwer, Jaak Simm, Adam Arany, Yves Moreau

arXiv:1905.12374v2cs.LGstat.ML

TL;DR

Sporadically observed multidimensional time series are difficult to model because measurements are irregular across time and variables. The paper combines continuous-time GRU dynamics with Bayesian updates for incoming observations. GRU-ODE-Bayes exactly represents relevant Fokker-Planck dynamics and outperforms other methods across synthetic and real-world evaluations, with continuity especially useful in small-sample settings.

  • Problem

    Multidimensional time series are often sampled irregularly across time and variables, challenging methods that assume systematic fixed-interval measurements.

    Source: §1 · p. 1

  • Method

    GRU-ODE-Bayes combines a continuous-time GRU with a Bayesian update network that processes sporadic observations within continuous ODE dynamics.

    Source: §1 · p. 2, §6 · p. 9

  • Results

    GRU-ODE-Bayes exactly represents Fokker-Planck dynamics for Ornstein-Uhlenbeck processes and outperforms the state of the art on healthcare and climate data.

    Source: §1 · p. 2

  • Takeaways & Limitations

    The continuity prior is particularly important in the small-sample regime relevant to modestly sized clinical datasets.

    Source: §6 · p. 9

    The work focuses on Gaussian observations, although the method can be extended to binomial and multinomial observations.

    Source: §6 · p. 9

Abstract

from arXiv · show

Modeling real-world multidimensional time series can be particularly challenging when these are sporadically observed (i.e., sampling is irregular both in time and across dimensions)-such as in the case of clinical patient data. To address these challenges, we propose (1) a continuous-time version of the Gated Recurrent Unit, building upon the recent Neural Ordinary Differential Equations (Chen et al., 2018), and (2) a Bayesian update network that processes the sporadic observations. We bring these two ideas together in our GRU-ODE-Bayes method. We then demonstrate that the proposed method encodes a continuity prior for the latent process and that it can exactly represent the Fokker-Planck dynamics of complex processes driven by a multidimensional stochastic differential equation. Additionally, empirical evaluation shows that our method outperforms the state of the art on both synthetic data and real-world data with applications in healthcare and climate forecast. What is more, the continuity prior is shown to be well suited for low number of samples settings.

1 Introduction

Real-world multidimensional time series are often sampled irregularly across time and variables, challenging methods built for fixed, systematic measurements. GRU-ODE-Bayes addresses this with continuous ODE dynamics tightly integrated with Bayesian processing of sporadic observations.

Motivation

  • Sporadic time series have irregular sampling times and missing variables at measurement events, violating assumptions of traditional recurrent methods.Clinical measurements illustrate this pattern because visits may be skipped and not every variable is recorded at each visit.Source: §1 · p. 1
  • Neural ODEs offer a continuous representation that can integrate dynamics over arbitrary time intervals, but prior uses largely generated observations through decoders or flows.The paper identifies a gap between continuous latent dynamics and direct processing of incoming sporadic observations.Source: §1 · pp. 1–2

Approach

  • GRU-ODE-Bayes interleaves continuous-time GRU dynamics with a Bayesian update network so incoming sporadic observations directly drive the latent dynamics.This tight integration replaces an encoder-decoder design in which ODE dynamics are decoupled from input processing.Source: §1 · p. 2

Results

  • Tight coupling enables fine-grained nonlinear interactions, including inferring an unobserved variable’s jump from a correlated observed variable.The method is reported to quickly infer underlying stochastic-process parameters and variable correlations, unlike NeuralODE-VAE’s more general process structure.Source: §1 · p. 2, Figure 1 · p. 2

Theory and evaluation

  • GRU-ODE-Bayes can exactly represent Fokker-Planck dynamics for Ornstein-Uhlenbeck processes and generalized versions, while outperforming the state of the art on healthcare and climate data.The paper frames the analysis around observations generated by multidimensional stochastic differential equations.Source: §1 · p. 2, §1.1 · p. 3

2 Proposed method

GRU-ODE-Bayes alternates continuous hidden-state evolution with Bayesian-style updates when sporadic observations arrive. Its design supports filtering, continuity, bounded hidden states, and end-to-end training from sporadically sampled data.

GRU-ODE-Bayes

  • The system combines GRU-ODE propagation between observations with GRU-Bayes updates when new observations become available.This creates an ODE with jumps at observation times.Source: §2 · p. 3, §2.4 · p. 4

GRU-ODE

  • GRU-ODE is derived from the GRU difference equation by taking a continuous-time limit for the hidden state.The resulting system is named GRU-ODE and has a minimal-GRU variant.Source: §2.1 · p. 3
  • GRU-ODE supports continuous inputs when available and becomes autonomous when no continuous input signal is provided.Clinical trial drug doses are given as an example of a continuous input.Source: §2.1 · p. 4

General properties of GRU-ODE

  • The hidden state remains within [-1, 1], with negative feedback stabilizing the system and correcting states that start outside this range.Boundedness makes the continuous dynamics compatible with GRU-Bayes and robust to numerical errors.Source: §2.2 · p. 4
  • GRU-ODE is Lipschitz continuous with constant K = 2, encoding a continuity prior for the latent process.The paper reports that this prior is empirically important in the small-sample regime.Source: §2.2 · p. 4

GRU-Bayes

  • GRU-Bayes preprocesses masked observations and updates the hidden representation, allowing adaptation within the GRU-ODE state region from a single observation.The update maps h(t−) to h(t+).Source: §2.3 · p. 4

Objective function

  • The model trains with pre-jump negative log-likelihood and post-jump KL-divergence losses computed on observed dimensions.The post-jump objective encourages the model to mimic a Bayesian update, with the two losses combined using λ.Source: Objective function · p. 5

Implementation

  • For minibatches, hidden states are propagated jointly while updates and losses run only for series observed at each time; complexity is linear in observations and quadratic in observation dimension.The adjoint method can reduce memory use by avoiding backpropagation through solver operations.Source: Objective function · p. 5

3 Related research

Prior work commonly handles sporadic time series through binning, imputation, probabilistic models, or continuous decoders. These approaches leave limitations in processing irregular inputs, forecasting, parameter identification, or continuous-time modeling.

Irregular observations

  • Fixed-duration binning creates missing observations across time and features, so recurrent architectures typically require imputation, masks, and observation times.The resulting representation makes direct neural-network usage difficult.Source: §3 · p. 6

Probabilistic models

  • Gaussian processes support imputation, latent-process modeling, representations, and forecasting, but usually have high uncertainty outside the observation range.The passage lists multitask Gaussian processes as the most popular probabilistic approach to missing data.Source: §3 · p. 6

Neural ODE approaches

  • Neural ODE–VAE methods naturally generate irregularly sampled data but transfer sporadic-data processing to a discrete-time recurrent encoder.A concomitant VAE architecture uses a Neural ODE, while the cited earlier approach couples Neural ODEs with a VAE.Source: §3 · p. 7

Sequential latent models

  • Discrete-time sequential-latent autoencoders use classical recurrent inference networks, making them less suited to sporadic data and forecasting.They have been useful for smoothing and counterfactual inference.Source: §3 · p. 7

State-space methods

  • Extended Kalman filters model continuous-time distribution dynamics, but linearized state updates and parameter-identification difficulties limit practical applicability.The related discussion also notes that GRUs can learn long-term dependencies.Source: §3 · p. 7

PDE-motivated architectures

  • PDE-motivated deep architectures generally result in discrete designs and do not explore continuous inputs and time.This distinguishes their motivation from continuous-time modeling of sporadic inputs.Source: §3 · p. 7

4 Application to synthetic SDEs

On synthetic stochastic systems, GRU-ODE-Bayes captures interactions between variables and refines predictions as observations arrive. The paper also shows exact representation for multivariate Ornstein–Uhlenbeck dynamics and demonstrates handling of a nonlinear Brusselator-inspired system.

Synthetic OU process

  • Compared with NeuralODE-VAE, GRU-ODE-Bayes detects correlations between features and updates predictions more finely as new observations arrive.NeuralODE-VAE retrieves average sample dynamics, whereas GRU-ODE-Bayes uses incoming observations to refine predictions and confidence.Source: §4 · p. 7
  • An observation of Dimension 2 updates the prediction for Dimension 1 at time t = 3.This illustrates cross-feature updating in the multivariate process.Source: §4 · p. 7

Filtering behavior

  • GRU-ODE-Bayes sequentially filters the hidden state and estimates the probability density of future observations, supporting long-term predictions.Sporadic inputs are fed directly into the ODE during this filtering process.Source: §4 · p. 7

Theoretical and nonlinear dynamics

  • The model exactly represents the dynamics of multivariate Ornstein–Uhlenbeck processes with random variables and also handles nonlinear SDEs in a Brusselator-inspired example.The Brusselator example is described as a chaotic ODE.Source: §4 · p. 7

5 Empirical evaluation

The model was evaluated for forecasting sporadic time series in healthcare and climate settings, using state-of-the-art baselines and cross-validation. GRU-ODE-Bayes outperformed the other methods on both datasets, with its continuity prior especially valuable when training samples were scarce.

Healthcare data

  • The healthcare task used 21,250 MIMIC-III patients and forecast the next three measurements after a 36-hour observation window.The cohort provided 96 longitudinal real-valued measurements over 48 hours after admission.Source: §5.2 · p. 8

Climate data

  • The climate task used sporadic observations from 1,114 USHCN stations across five climate variables over a four-year window.The source data covered daily measurements collected over 150 years from stations across the United States.Source: §5.3 · p. 8

Results

  • GRU-ODE-Bayes unequivocally outperformed all other methods on both the healthcare and climate datasets.Performance was assessed using negative log-likelihood and mean squared error with 5-fold cross-validation.Source: §5 · p. 7, §5.4 · p. 8

Impact of continuity prior

  • The continuity prior is particularly important when only a small number of samples is available.The paper frames this setting as relevant to rare-disease prediction, where data may be available for few patients.Source: §5.5 · pp. 8–9
  • In the small-sample MIMIC-III regime, GRU-ODE achieved higher accuracy than the discretized GRU version.The discretized model matched the continuous model on the full dataset, whereas the continuous cell performed better with fewer training patients.Source: §5.5 · pp. 8–9, Table 2 · p. 9

6 Conclusion and future work

The paper combines continuous-time hidden-state evolution with Bayesian updates for sporadic observations and establishes continuity and representation properties. It reports strong performance on synthetic and real-world data while noting Gaussian observations as the primary scope.

Conclusion

  • GRU-ODE-Bayes combines GRU-ODE and GRU-Bayes to feed sporadic observations into continuous ODE dynamics for the data distribution.The system evolves the hidden state between observations and updates it when observations arrive.Source: §6 · p. 9

Theoretical properties

  • The model can exactly represent Fokker-Planck dynamics for Ornstein-Uhlenbeck processes and generalized versions driven by stochastic differential equations.This is presented as a theoretical representation capability of the proposed filtering approach.Source: §6 · p. 9

Future relevance

  • The continuity prior is particularly important in the small-sample regime relevant to modest-sized clinical datasets.The discretized model performed well on the full MIMIC-III dataset, but the continuous formulation was more valuable with fewer samples.Source: §6 · p. 9

Scope

  • The experiments focused on time-series data with Gaussian observations, although the method can be extended to binomial and multinomial observations.The paper states that the corresponding negative log-likelihood and KL-divergence are analytically tractable for those extensions.Source: §6 · p. 9

Continuity

  • GRU-ODE is Lipschitz continuous with constant K = 2, encoding a continuity prior for the latent process.The bound follows from bounded hidden states and bounded sigmoid and hyperbolic-tangent functions in the ODE.Source: B Lipschitz continuity of GRU-ODE · p. 13

C Comparison of numerical integration methods

The implementation compares Euler, explicit midpoint, and Dormand-Prince integration for the continuous GRU-ODE. Adaptive integration reduces the number of steps, while Euler remains computationally competitive and DOPRI offers greater numerical stability.

Integration methods

Step allocation

Computational trade-off

Observation processing

H Ablation study of GRU-Bayes

The study compares the GRU-Bayes module with an MLP replacement and evaluates GRU-ODE-Bayes on correlated Ornstein–Uhlenbeck processes under varying parameters, lags, and correlations.

Ablation study of GRU-Bayes

  • Replacing GRU-Bayes with a two-layer MLP produces worse performance on the MIMIC forecasting task.The comparison uses a tanh hidden activation and linear output activation.Source: H Ablation study of GRU-Bayes · p. 15

Representation capabilities

  • GRU-ODE-Bayes exactly represents the Fokker–Planck dynamics of the OU process when its distribution is Gaussian with time-dependent mean and covariance.The analysis is conditioned on a previous observation and expresses the distributional evolution through mean and variance dynamics.Source: §I.1 · pp. 15–16
  • GRU-ODE-Bayes models generalized Ornstein–Uhlenbeck processes by evolving the hidden state between observations and updating it when sporadic observations arrive.GRU-ODE stores continuous-time dynamics, while GRU-Bayes updates estimates such as sample-dependent parameters.Source: §I.1.2 · p. 16

Experimental setup

  • The OU evaluation varies target parameters across samples, adds random time lags, and compares highly correlated dimensions with independent dimensions.The correlated setup uses ρ = 0.99, while the independent setup uses ρ = 0.Source: §I.2.1 · p. 16
  • The training set contains 10,000 samples with an average of 20 observations scattered over 10 seconds, and forecasting is evaluated after t = 4.Methods are trained with negative log-likelihood, with MSE also reported, and compared against NeuralODE-VAE variants.Source: §I.2.1 · pp. 16–17

I.2.2 Empirical evaluation

GRU-ODE-Bayes updates predictions sequentially as sporadic observations arrive, using correlations between dimensions to refine unobserved variables and confidence estimates.

Empirical evaluation

  • GRU-ODE-Bayes updates a feature’s prediction and confidence even when only the correlated other feature is observed.For example, Dimension 1 is updated when Dimension 2 is observed at t = 3.Source: §I.2.2 · p. 17
  • Sequentially filtering the hidden state lets GRU-ODE-Bayes estimate the future-observation PDF and perform long-term predictions.This differs from NeuralODE-VAE’s strategy of storing the whole dynamics in a single vector before mapping it to the time series.Source: §I.2.2 · p. 17
  • GRU-ODE-Bayes performs better than the compared methods on all tested setups for both NegLL and MSE.Random time lags cause only marginal degradation in NegLL and MSE.Source: §I.2.2 · p. 17
  • When the dimensions are uncorrelated (ρ = 0), observing one dimension provides no information about the other and performance is lower.The result follows the absence of shared information between the dimensions.Source: §I.2.2 · p. 18
  • As more observations are processed, confidence intervals narrow and predictions converge toward the underlying process’s asymptotic distribution.The initial predictions have broad intervals and progressively refine the variance estimate.Source: §I.2.2 · p. 18, Figure 5 · p. 18

J Application to synthetic nonlinear SDE: the Brusselator

The paper evaluates GRU-ODE-Bayes on a nonlinear stochastic Brusselator process and shows that its filtered predictions use observations to share information across dimensions.

Application to synthetic nonlinear SDE: the Brusselator

L.1 Cleaning and subsampling

The climate-data preprocessing removes poor-quality records, restricts the time period and centers, subsamples observations, and retains the final four years for analysis.

Cleaning and subsampling

  • After cleaning and subsampling, the USHCN dataset contains 1,114 centers and 386,068 unique observations.The observations are sporadically distributed over the final four years, averaging 346 observations per center.Source: §L.1 · p. 20

M Small-sample regime: additional results

Additional MIMIC-III results show that the continuity prior strongly helps GRU-ODE-Bayes in the low-sample regime, with performance reported using negative log-likelihood. The evaluation used held-out testing, five-fold cross-validation, and shared folds across models.

Small-sample results

Case-study evaluation

Experimental setup

  • Models used the same hidden-state dimension, with hyperparameters tuned on a 20% validation split for each fold.The tuned parameters included dropout, weight decay, and learning rate.Source: O Hyper-parameters used · pp. 21–22
  • Performance was assessed on a held-out test set using five-fold cross-validation, with average and standard-deviation results reported across folds.The same folds were reused for each compared model to support reproducible and fair comparisons.Source: O Hyper-parameters used · p. 22

Related papers

Loading 1905.12374v2…