PaperlayerView exact revision on arXiv

Source-linked AI summary

Modeling Irregular Time Series with Continuous Recurrent Units

Mona Schirmer, Mazin Eltayeb, Stefan Lessmann, Maja Rudolph

arXiv:2111.11344v3cs.LGstat.ML

TL;DR

Irregular observation times challenge modern RNNs because timing can carry information. The paper introduces CRU, which combines a continuous latent-state model with Kalman-filter updates in an encoder-decoder architecture. Across challenging datasets, CRU improves interpolation over neural ODE-based methods, while f-CRU reduces training time substantially.

  • Problem

    Irregularly sampled data challenge modern RNNs because observation timing can carry information about the underlying time series.

  • Method

    CRU uses an encoder-decoder architecture with a latent state governed by a linear SDE and closed-form continuous-discrete Kalman-filter recursion.

  • Results

    CRU better interpolates irregular time series than neural ODE-based models and outperforms established recurrent models such as GRU on irregularly sampled data.

  • Takeaways & Limitations

    CRU’s uncertainty-aware gating accurately weights noisy and partially observed inputs, providing temporal continuity between hidden states.

  • Takeaways & Limitations

    The method assumes a linear SDE with a Gaussian observation model, while its diffusion matrix is fixed to the identity in one parameterization.

Abstract

from arXiv · show

Recurrent neural networks (RNNs) are a popular choice for modeling sequential data. Modern RNN architectures assume constant time-intervals between observations. However, in many datasets (e.g. medical records) observation times are irregular and can carry important information. To address this challenge, we propose continuous recurrent units (CRUs) -- a neural architecture that can naturally handle irregular intervals between observations. The CRU assumes a hidden state, which evolves according to a linear stochastic differential equation and is integrated into an encoder-decoder framework. The recursive computations of the CRU can be derived using the continuous-discrete Kalman filter and are in closed form. The resulting recurrent architecture has temporal continuity between hidden states and a gating mechanism that can optimally integrate noisy observations. We derive an efficient parameterization scheme for the CRU that leads to a fast implementation f-CRU. We empirically study the CRU on a number of challenging datasets and find that it can interpolate irregular time series better than methods based on neural ordinary differential equations.

1. Introduction

Modern RNNs struggle with irregularly sampled data because observation timing can carry information, motivating CRU, a continuous probabilistic recurrent architecture with efficient implementation and broad empirical evaluation.

  • Motivation: Irregular observation times can encode information about underlying time series, but modern discrete RNNs typically assume regular sampling.Examples include medical records and climate data, where time between observations may be informative.
  • Proposed approach: CRU models irregularly sampled sequences using a probabilistic latent state governed by a linear stochastic differential equation.Its recursive computations use the continuous-discrete Kalman filter.
  • Proposed approach: The continuous-discrete Kalman-filter formulation provides uncertainty for gating and confidence evaluation, locally optimal filtering, and analytic latent-state resolution.Analytic resolution avoids numerical integration techniques or variational approximations.
  • Contributions: CRU combines neural feature extraction with a continuous-discrete Kalman filter to model data with irregular observation times.The paper also provides a PyTorch implementation.
  • Contributions: The eigendecomposition-based transition-matrix parameterization yields a faster implementation called fast CRU, or f-CRU.This contribution targets computational efficiency.
  • Empirical evaluation: Across electronic health records, climate data, and images, CRU better interpolates irregular time series than neural ODE-based methods.The CRU also handles uncertainty from noisy and partially observed inputs, while outperforming discrete RNNs and neural ODE models on image data.

2. Related Work

Related work includes stochastic, irregular-time, continuous-time, neural ODE/SDE, and attention-based sequence models, with CRU positioned as a continuous probabilistic alternative.

  • Stochastic RNNs: Stochastic RNNs address variability in data that deterministic internal transitions may fail to capture, including through deep probabilistic Kalman-filter approaches.The passage situates CRU among stochastic recurrent models.
  • RNNs for Irregular Time Series: Discretizing irregular time series into uniform bins can reduce observations, lose information, and require imputation or aggregation.Timestamp and observation-mask approaches avoid some preprocessing but lack dynamics between observations.
  • RNNs for Irregular Time Series: Exponential hidden-state decay models handle elapsed time but are limited to decaying dynamics, whereas CRU is more expressive.This contrasts CRU with a restricted irregular-time strategy.
  • Continuous-Time RNNs: Continuous-time RNNs govern internal units with systems of ODEs and have roots in early recurrent-network research.The cited work also developed gradient-based optimization theory for their parameters.
  • Neural ODEs: Neural ODE models evolve hidden states continuously, while ODE-RNN and ODE-LSTM add observation-time recurrent updates.Latent ODE lacks an update mechanism for incorporating incoming observations into its latent trajectory.
  • Neural SDEs: Neural SDEs use stochastic differential equations for latent temporal processes, covering generative, discontinuous, flow-based, and adversarial approaches.The passage identifies neural SDEs as stochastic counterparts to neural ODEs.
  • Transformers for Irregular Time Series: Attention-based methods model arbitrary timestamps using time-gap decay, set functions, or time embeddings.These methods form another alternative to recurrent and differential-equation architectures.

3. Method

The CRU models irregularly sampled time series by evolving a continuous latent state with a linear SDE and recursively applying continuous-discrete Kalman filtering within an encoder-decoder architecture. Its f-CRU variant reduces computation through a restricted transition-matrix parameterization, trading modeling flexibility for speed.

  • 3.1. Overview of Proposed Approach: The CRU maps observations into a latent space whose continuous state evolves according to a linear SDE with Gaussian observation noise.The model uses time-invariant transition and diffusion parameters and distinguishes prior and posterior distributions at observation times.
  • 3.2. Continuous-Discrete Kalman Filter: At each observation time, the CRU alternates an analytical prediction step with a Bayesian update step derived from the continuous-discrete Kalman filter.The prediction propagates the latent state between observations, while the update incorporates the new observation into the posterior.
  • 3.2. Continuous-Discrete Kalman Filter: The Kalman gain acts as a gate that weights the prior against each observation according to observation noise and prior uncertainty.The resulting posterior mean and covariance are maintained as the CRU cell’s internal state and are decoded into outputs with elementwise uncertainties.
  • 3.3. Recurrent Architecture: CRU recursion is deterministic, closed form, differentiable, and trainable end-to-end like standard recurrent architectures while handling irregular observation times.The encoder-decoder supplies nonlinear feature extraction and output mapping around the probabilistic latent-state computations.
  • 3.4.2. Efficient Implementation: The f-CRU restricts transition matrices to commutative, symmetric forms, replacing the matrix exponential’s O(n^3) computation with matrix multiplication and elementwise operations.This restriction reduces runtime but limits modeling flexibility; the parameterization is intended to support larger latent state spaces.
  • 3.4.2. Efficient Implementation: The f-CRU parameterization yields computational gains that become especially relevant as latent dimensionality increases, motivating a speed–accuracy comparison with the CRU.The paper studies this trade-off empirically and describes the efficient prior computation in Appendix A.1.2.

4. Empirical Study

The empirical study evaluates CRU and f-CRU on interpolation, regression, and extrapolation tasks using irregular or sparse sequence data from pendulum images, climate records, and electronic health records. CRU methods generally improve accuracy or runtime relative to baseline sequence models, while the Kalman-gain mechanism adapts updates to noisy and partially observed inputs.

  • Experimental setup: The study compares CRU and f-CRU with discrete RNNs, neural ODEs, and attention-based models across interpolation, regression, and extrapolation tasks.Experiments use pendulum images, USHCN climate data, and Physionet electronic health records.
  • Runtime: CRU trains consistently faster than neural ODE-based models because its closed-form propagation avoids ODE solvers and its batching cost does not scale with timestamp heterogeneity.This runtime advantage is particularly relevant when timestamps vary widely across sequences such as Physionet.
  • Sequence interpolation: Both CRU variants outperform baseline models on most interpolation datasets, while f-CRU reduces training time by up to 50% with comparable results.Discrete RNNs remain faster, but f-CRU takes only a fraction of the training time of neural ODE-based methods.
  • Uncertainty handling: The Kalman gain assigns less weight to observations under high noise and greater weight when the pendulum is clearly observed.For partially observed USHCN inputs, higher sparsity shifts the Kalman-gain distribution toward lower norm values.
  • Pendulum angle prediction: CRU outperforms existing baselines on pendulum angle prediction, and CRU and f-CRU achieve the best reported log-likelihood performance.The pendulum regression task uses images corrupted by a correlated noise process.
  • Sequence extrapolation: Extrapolation results vary by dataset: mTAND achieves the lowest Physionet errors, whereas CRU reaches the highest performance on climate data.The extrapolation task predicts the remaining sequence from its first half and therefore requires long-term temporal interactions.

5. Conclusion

CRU models non-uniform time intervals by combining an encoder-decoder with a continuous-discrete Kalman filter, giving hidden states temporal continuity and uncertainty. Its efficient implementation uses structured transition and covariance computations, while simplified Kalman updates reduce matrix operations.

  • 5. Conclusion: CRU models temporal data with non-uniform time intervals using a continuous-discrete Kalman filter within an encoder-decoder architecture.This introduces temporal continuity into the hidden state and a notion of uncertainty.
  • 5. Conclusion: CRU assumes locally linear transitions with symmetric basis matrices sharing eigenvectors, enabling efficient matrix-exponential computations.Under these assumptions, the matrix exponential can be reduced to elementwise exponentiation after diagonalization.
  • 5. Conclusion: The prior covariance is computed analytically using eigendecomposition or matrix fraction decomposition rather than numerical integration.The matrix fraction approach computes the required integral by solving a matrix-valued ODE and evaluating a matrix exponential.
  • 5. Conclusion: Structured covariance assumptions make the Kalman gain and posterior updates reducible to simpler elementwise operations.The observation and memory components are correlated only through corresponding blocks, supporting the simplified update equations.
  • 5. Conclusion: The latent state uses an observation part for direct inputs and a memory part for features inferred over time.The observation model fixes H = [I_D, 0_D] and sets M = 2D.

C. Log-likelihood Results

The pendulum regression evaluation reports Gaussian negative log-likelihood on test data to assess uncertainty under high observation noise. CRU outperforms baseline models on this uncertainty metric.

  • C. Log-likelihood Results: Gaussian NLL on test pendulum-regression data evaluates uncertainty computation under high observation noise.The reported values are presented as mean ± standard deviation.
  • C. Log-likelihood Results: CRU outperforms baseline models on the pendulum regression NLL evaluation.

D. Implementation Details

The experiments train each model with a common optimization schedule while using dataset-specific batching and validation splits. Pendulum results use repeated generated sequences and a held-out test set.

  • D. Implementation Details: All experiments train each model for 100 epochs with the Adam optimizer.Reported MSE and Gaussian NLL results are averages over 5 runs.
  • D. Implementation Details: Batch sizes are 50 for pendulum and USHCN data and 100 for Physionet.USHCN and Physionet use 80% train and 20% test splits, with 25% of the training set reserved for validation.
  • D. Implementation Details: Pendulum experiments use 2,000 training sequences, 1,000 validation sequences, and a 1,000-sequence hold-out test set.The same folds are reused for each compared model.
  • D. Implementation Details: Hyperparameters are selected using prior published choices for GRU-D, latent ODE, ODE-RNN, and mTAND configurations.The procedure covers latent-state size, layer count, hidden units, and training parameters.

D.1. mTAND

The implementation uses dataset-specific hidden-state sizes and adapts baseline architectures for comparison. Time-aware variants receive observation gaps as additional inputs to their transition or recurrent networks.

  • D.1. mTAND: Hidden-state sizes are M = 30 for pendulum, M = 20 for Physionet, and M = 10 for USHCN.The CRU architecture and hyperparameter choices are kept consistent across experiments.
  • D.1. mTAND: mTAND-Full uses 10 hidden units in its encoder and decoder and has three times as many parameters as CRU.mTAND-Enc uses 10 hidden units for the per-time-point regression task.
  • D.1. mTAND: RKN-∆t receives time gaps as additional input to its transition network, which learns weights α_τ(t) for the basis matrices.
  • D.1. mTAND: GRU and GRU-∆t use hidden-state size 75 on all datasets to make parameter sizes comparable.GRU-∆t concatenates the time gap between observations with the recurrent-cell input.

D.4. GRU-D

The GRU-D configuration uses dataset-specific hidden-state sizes while keeping other training settings fixed across experiments.

  • Hidden-state sizes are M = 30 for pendulum, M = 20 for Physionet, and M = 10 for USHCN.
  • The experiments use 100 hidden units and a learning rate of 0.01 with decay factor 0.999.
  • Other hyperparameter and architecture choices follow Rubanova et al. (2019).

D.5. Latent ODE

The Latent ODE comparison uses dataset-specific latent dimensions and prescribed architectures, while CRU and f-CRU are trained with separate learning rates selected on pendulum interpolation.

  • Latent ODE configuration: The latent ODE uses latent state sizes M = 30 for pendulum, M = 20 for Physionet, and M = 10 for USHCN.
  • Latent ODE configuration: Its recognition model has 40 hidden dimensions, while its ODE and GRU components use 50 hidden units.
  • Training: CRU uses Adam and Gaussian negative log-likelihood for Physionet, USHCN, and pendulum angle prediction, but uses Eqn. (19) for pendulum interpolation.
  • Training: The selected learning rates are 0.001 for CRU and 0.005 for f-CRU across the experiments.
  • CRU architecture: The f-CRU initialization sets basis matrices to zero, with E = I and D(k) = 1e−5 · I, to align the prior mean with the previous posterior mean.
  • Pendulum interpolation: The pendulum interpolation CRU uses latent observation dimension 15, latent state dimension 30, 15 basis matrices, and bandwidth 3.

E. Data Preprocessing

The experiments use publicly available weather and clinical datasets with dataset-specific filtering, time handling, normalization, and train-validation-test splits.

  • USHCN: USHCN processing removes observations with bad quality flags and selects a 1990–1993 window for long-term extrapolation.
  • USHCN: The USHCN data retains 1168 centers that report across the selected period and splits them into 60% train, 20% validation, and 20% test sets.
  • PhysioNet: PhysioNet preprocessing discards four admission-time descriptors and retains 37 time-variant features rounded to six-minute steps.
  • PhysioNet: PhysioNet patients are split into 60% train, 20% validation, and 20% test sets, with each feature normalized separately to [0,1].
  • Compute: The models are trained on an Nvidia TU102GL Quadro RTX 6000/8000 system with 40 physical Intel Xeon Gold 6242R CPUs.

H. Sample Trajectory

The sample-trajectory figures present the f-CRU architecture and a pendulum interpolation trajectory under irregular image-frame intervals.

  • Figure 6 presents the f-CRU architecture.
  • Figure 7 shows a test trajectory for the pendulum interpolation task.
  • The figure caption states that f-CRU predicts images precisely despite irregular intervals between image frames.
Loading 2111.11344v3…