PaperlayerView exact revision on arXiv

Source-linked AI summary

Latent ODEs for Irregularly-Sampled Time Series

Yulia Rubanova, Ricky T. Q. Chen, David Duvenaud

arXiv:1907.03907v1cs.LGstat.ML

TL;DR

Irregularly sampled time series challenge standard RNNs because preprocessing can discard measurement-timing information. The paper develops ODE-RNNs and uses them to encode Latent ODEs, enabling continuous-time modeling with arbitrary observation gaps; experiments show advantages over RNN counterparts on irregular data.

  • Problem

    Standard RNNs are awkward for irregularly sampled time series, while equal-time binning and imputation can destroy informative measurement timing.

  • Method

    The paper defines ODE-RNN hidden dynamics with neural ODEs and uses ODE-RNNs as both standalone autoregressive models and Latent ODE recognition networks.

  • Results

    ODE-based models outperform their RNN-based counterparts on irregularly sampled data, with Latent ODEs outperforming standard RNN-VAEs on interpolation and extrapolation.

  • Takeaways & Limitations

    ODE-RNNs and Latent ODEs handle arbitrary observation gaps without discretizing observation times or imputing data, while Latent ODEs provide explicit uncertainty estimates.

Abstract

from arXiv · show

Time series with non-uniform intervals occur in many applications, and are difficult to model using standard recurrent neural networks (RNNs). We generalize RNNs to have continuous-time hidden dynamics defined by ordinary differential equations (ODEs), a model we call ODE-RNNs. Furthermore, we use ODE-RNNs to replace the recognition network of the recently-proposed Latent ODE model. Both ODE-RNNs and Latent ODEs can naturally handle arbitrary time gaps between observations, and can explicitly model the probability of observation times using Poisson processes. We show experimentally that these ODE-based models outperform their RNN-based counterparts on irregularly-sampled data.

1 Introduction

Irregular sampling makes standard RNNs awkward because binning and imputation can destroy informative measurement timing. The paper introduces continuous-time ODE-based models that evolve hidden states between observations while updating them at observation times.

  • Motivation: Irregularly sampled time series are difficult for standard RNNs, and equal-time binning with imputation or averaging can destroy informative measurement timing.Timing can contain information about latent variables.
  • Approach: ODE-RNN states obey an ODE between observations and are updated when observations arrive, unlike standard RNN states that are constant or undefined between observations.Figure 1 contrasts these trajectories with exponentially decaying RNN-Decay states and Neural ODE states determined by their initial state.
  • Approach: ODE-RNNs generalize RNN state transitions by specifying continuous-time hidden dynamics with a neural network, following the Neural ODE approach.The hidden state evolves according to an ODE between observations.
  • Approach: The paper uses ODE-RNNs both as standalone autoregressive models and as recognition networks for Latent ODE models.Latent ODEs generate time series from the deterministic evolution of an initial latent state and can be trained as variational autoencoders.
  • Implications: ODE-based models naturally handle arbitrary time gaps without grouping observations into equal-time bins, and Latent ODEs can model observation times with a Poisson process.The paper reports better ODE-RNN performance when data is sparse and jointly models observation-time probabilities in Latent ODEs.
  • Background: Neural ODE hidden states are defined at all times and can be evaluated at desired times using a numerical ODE solver.The dynamics function fθ is represented by a neural network.

3 Method

The paper constructs continuous-time autoregressive and latent-variable models by combining neural ODE dynamics with observation-driven updates. Latent ODEs use an ODE-RNN encoder, can model observation times with a Poisson process, and offer explicit latent states and uncertainty.

  • 3.1 Constructing an ODE-RNN Hybrid: ODE-RNNs evolve the hidden state with a neural ODE between observations, then apply an RNN update when each observation arrives.This replaces fixed or exponentially decayed between-observation dynamics with a more flexible parameterization.
  • Autoregressive Modeling with the ODE-RNN: The ODE-RNN can define autoregressive conditional distributions for one-step-ahead sequence prediction.As in standard RNNs, it factors the joint density into conditionals based on the observation history.
  • 3.2 Latent ODEs: a Latent-variable Construction: Latent ODEs sample an initial latent state, evolve it deterministically with an ODE, and generate observations conditionally from the resulting latent states.This is trained and used within a variational autoencoder framework.
  • 3.2 Latent ODEs: a Latent-variable Construction: An ODE-RNN encoder runs backward through irregular observations, and its final hidden state is mapped to the mean and variance of the approximate posterior over the initial state.The resulting encoder-decoder architecture handles variable-length input sequences and produces a fixed-dimensional latent representation.
  • 3.2 Latent ODEs: a Latent-variable Construction: The latent-variable framework decouples system dynamics, observation likelihood, and recognition, while providing explicit uncertainty over latent states.It also supports non-standard queries such as backward-time prediction or conditioning on subsets of observations.
  • 3.3 Poisson process likelihoods: Latent ODEs can jointly model observations and their times by parameterizing a time-varying Poisson intensity λ(t) from the latent trajectory.The joint likelihood evaluates the intensity, its integral, and required latent states using an ODE solver.
  • 3.4 Batching and computational complexity: ODE-based models have similar asymptotic time complexity to standard RNNs, but ODE-RNN evaluation took 60% more time than a standard GRU and Latent ODE evaluation roughly twice as long as ODE-RNN evaluation.Their compute cost does not scale with data sparsity because the ODE is solved even without observations.

4 Experiments

Experiments evaluate ODE-RNNs and Latent ODEs on toy periodic trajectories, MuJoCo physics, and clinical time series, emphasizing sparse observations, extrapolation, uncertainty, and observation-time modeling.

  • Toy dataset: The toy dataset contains 1,000 noisy periodic trajectories with variable frequency, 100 irregularly sampled points, and random subsampling during training.Models reconstruct the full trajectory from a fixed subset of observations.
  • Toy dataset: Conditioning on more points improves Latent ODE reconstruction and reduces predictive variance, while prior samples remain periodic.The model was trained with 30 points and tested with subsets of 10, 30, or 50 points.
  • Toy dataset: An ODE-RNN recognition network extrapolates periodic dynamics far beyond the training interval, whereas an RNN recognition network does not extrapolate them well.Both models condition on 20 points in [0, 2.5] and predict over [2.5, 5].
  • MuJoCo Physics Simulation: On MuJoCo, Latent ODEs outperform standard RNN-VAEs on interpolation and extrapolation, while ODE-RNNs outperform standard RNNs on interpolation, especially with sparse data.The Latent ODE also performs better than the autoregressive ODE-RNN.
  • MuJoCo Physics Simulation: All autoregressive models perform poorly at extrapolation, although standard RNNs perform better than ODE-RNNs in that setting.The authors attribute this to training only for one-step-ahead prediction.
  • MuJoCo Physics Simulation: In MuJoCo reconstructions, the latent dynamics norm spikes when the hopper hits the ground and is small when it lies on the ground.Posterior entropy decreases as more observations are provided, and latent states correspond closely to key physical parameters.
  • Physionet: ODE-based models have smaller MSE than RNN baselines on the evaluated dataset, while ODE-RNN, Latent ODE, and GRU-D achieve similar mortality-classification AUC.Adding a Poisson-process likelihood models observation times reasonably well but does not improve classification accuracy.

5 Related work

Related work adapts RNNs to irregular sampling through imputation, time-gap inputs, decay dynamics, interpolation, Gaussian processes, and event-intensity models.

  • Irregular-sampling RNNs: A common strategy discretizes time, imputes missing observations, and applies an RNN, using weighted averages, interpolation networks, or Gaussian processes.These methods accommodate irregular sampling through preprocessing or auxiliary models.
  • Time-gap dynamics: Other approaches append Δt to RNN inputs or impose exponential hidden-state decay, but prior work found that these changes did not necessarily improve predictive performance.Appending Δt was reported as prone to overfitting, while exponential-decay dynamics did not outperform standard RNNs empirically.
  • Event-process models: Neural Hawkes processes explicitly model observation intensities with self-exciting latent states, whereas Latent ODEs treat observations as updating posterior beliefs without affecting latent dynamics.The distinction makes Latent ODEs more appropriate when observations do not alter the underlying system state.

6 Discussion and conclusion

The paper introduces ODE-RNNs with Neural ODE hidden dynamics and uses them to improve Latent ODE recognition networks for irregularly sampled data. These models avoid discretizing observation times, provide interpretable latent states and uncertainty estimates, and can model observation rates with Poisson processes.

  • ODE-RNNs specify hidden-state dynamics with neural ordinary differential equations.
  • The model serves both as a standalone RNN refinement and as a recognition network within a variational-autoencoder Latent ODE model.
  • Neither ODE-RNNs nor Latent ODEs requires discretizing observation times or imputing data during preprocessing.
  • Latent ODEs provide relatively interpretable latent states and explicit uncertainty estimates about those states.
  • Continuous-time latent states can be combined with Poisson-process likelihoods to model observation rates.

1 Experiment setup

The experiments evaluate ODE-based models on interpolation and extrapolation, using partially observed time series and task-specific choices of the ODE's initial time point. The setup tests reconstruction of missing or future observations.

  • The ODE models are evaluated on two tasks: interpolation and extrapolation.
  • Interpolation: Interpolation conditions on a subset of observations and reconstructs the full series within the same time interval.
  • Interpolation: In interpolation experiments, MUJoCo observations are subsampled at proportions ranging from 10% to 50%.
  • Extrapolation: Extrapolation encodes the first half of a series and reconstructs its second half from a randomly sampled subset of time points.
  • The generative model's initial time point is chosen at the first observation for interpolation, while the extrapolation task uses a different initial point.

2 Model details

The model combines latent ODE dynamics, neural-network transformations, and Poisson-process components to represent irregularly sampled time series. Latent states are evolved with an ODE solver and decoded into observations.

  • The latent initial state is sampled from an approximate posterior, then ODESolve evolves it across the requested time points.
  • Latent ODE: Latent states are transformed by an output network to produce reconstructed observations.
  • Modelling poisson process likelihood: The Poisson-process extension adds latent dimensions zλ and maps them through a neural network to observation intensities λ.
  • Modelling poisson process likelihood: The augmented ODE includes the intensity integral and is solved with a call to ODESolve.
  • Modelling poisson process likelihood: The joint generative model represents the initial latent state, observation times, and observations through p(z0)p(t0, . . . , tN|z0) ∏N.
  • The latent ODE framework defines a generative model for time series.

3 Data generation and preprocessing

The study uses synthetic and real-world time-series datasets with task-specific sampling, segmentation, and normalization procedures. The datasets cover trajectories, ICU measurements, and human activities.

  • Toy dataset: The toy dataset contains 1,000 one-dimensional sinusoidal trajectories with 100 time points over [0, 5].
  • MuJoCo: MuJoCo provides 10,000 Hopper simulations, with 200 time steps for extrapolation and 100 time points for interpolation.
  • PhysioNet: PhysioNet contains up to 37 features from the first 48 ICU hours, with timestamps rounded to one minute and up to 2,880 points per series.
  • Human Activity: Human Activity data are assembled into 6,554 sequences of 211 time points and reduced to seven activity classes.
  • The experiments use an 80% training and 20% test split, rescale PhysioNet features to [0, 1], and rescale timelines to [0, 1].

4 Architecture

The paper specifies neural ODE functions, numerical solvers, losses, and RNN baselines for its continuous-time models.

  • ODE functions use feed-forward neural networks with Tanh activations to constrain outputs and avoid excessively large ODE gradients.The authors do not recommend ReLU because large gradients can make ODE solving difficult at the specified tolerance.
  • Latent ODE generative models use the fifth-order adaptive-step dopri5 solver with relative tolerance 1e-3 and absolute tolerance 1e-4.The adjoint method can reduce memory use, but increases computation time.
  • Training uses fixed-variance negative Gaussian log-likelihood for reconstruction, reports test-set MSE, and uses cross-entropy for classification.ELBO computation uses three samples from N(µz0, σz0); Physionet training combines reconstruction and classification losses.
  • RNN baselines include GRU-D, exponential decay between hidden states, and imputation-based variants.

5 Hyperparameters

Hyperparameters are selected to favor strong RNN baselines and then matched across corresponding ODE models, with model dimensions varying by dataset and task.

  • Hyperparameter selection: ODE models use the best-performing RNN-baseline hyperparameters, matching hidden-state size, layer count, and encoder-decoder units where applicable.
  • Latent ODE model: Latent ODEs generally use generative ODEs at or slightly below data dimensionality, with recognition models assigned larger dimensionality.
  • Toy dataset: The toy dataset uses 10 generative and 20 recognition latent dimensions, batch size 50, and one 100-unit hidden layer in each ODE function.
  • MuJoCo: MuJoCo uses 15 generative and 30 recognition latent dimensions, batch size 50, three 500-unit ODE-function layers, and 15 autoregressive hidden units.
  • Physionet: Physionet uses 20 generative and 40 recognition latent dimensions, batch size 50, three 50-unit ODE-function layers, and 20 autoregressive hidden units.
  • Poisson process: Poisson-process modeling adds 20 dimensions for z(t), 20 for zλ, and 37 for the observation process, totaling 77 dimensions.The observation-process dimensionality equals the data dimensionality.
  • Classification: Physionet mortality classification combines reconstruction and cross-entropy losses, using coefficient 100 on cross-entropy and a two-layer 300-unit classifier.Reconstruction uses all 8000 patients, while cross-entropy uses 4000 labeled patients.
  • Classification: Human Activity classification uses a linear classifier on each hidden state, with task-specific autoregressive and encoder-decoder dimensions and ODE networks.

6 Training details

Training uses Adamax with learning-rate decay and KL annealing, runs on specified hardware, and evaluates models across several datasets and reconstruction settings.

  • Optimization: Optimization uses Adamax at learning rate 0.01, decay rate 0.999, and KL annealing with coefficient 0.99 for VAE models.
  • Compute: All experiments run on one Nvidia P100 GPU with two physical Intel Xeon Silver 4110 CPUs.
  • Implementation: Dataset preprocessing and generation code accompanies the submission, with models implemented in PyTorch 1.0 and ODE solvers from torchdiffeq.
  • Datasets: Experiments use Human Activity, Physionet, and MuJoCo datasets, with MuJoCo generated using DeepMind Control Suite.
  • Computational behavior: Figure 3 reports that ODE-function evaluations are independent of the number of evaluated time points but depend on the solved interval length.
  • Visualizations: Figures cover toy-data reconstructions, latent recognition paths, posterior distributions, MuJoCo trajectories, and inferred Physionet Poisson rates.
Loading 1907.03907v1…