PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Controlled Differential Equations for Irregular Time Series

Patrick Kidger, James Morrill, James Foster, Terry Lyons

arXiv:2005.08926v2cs.LGstat.ML

TL;DR

Irregularly sampled and partially observed time series challenge neural ODEs because later observations cannot adjust trajectories set by initial conditions. This paper introduces Neural CDEs, which process such data with memory-efficient adjoint training and achieve state-of-the-art performance across benchmark datasets.

  • Problem

    Neural ODE trajectories are fixed by their initial conditions and lack a mechanism to incorporate later observations, while discretisation breaks down for irregularly sampled or partially observed data.

  • Method

    The paper introduces Neural CDEs, continuous-time models that process irregularly sampled, partially observed multivariate series and support adjoint-based backpropagation across observations.

  • Results

    Neural CDEs achieve state-of-the-art performance across CharacterTrajectories, PhysioNet sepsis prediction, and Speech Commands, outperforming every considered model on CharacterTrajectories.

  • Takeaways & Limitations

    Neural CDEs combine direct handling of irregular and partially observed data, state-of-the-art performance, memory-efficient training, universal approximation, and subsumption of alternative ODE models.

  • Takeaways & Limitations

    Natural cubic spline paths are noncausal, making Neural CDEs infeasible for real-time settings because the path can depend on future observations.

Abstract

from arXiv · show

Neural ordinary differential equations are an attractive option for modelling temporal dynamics. However, a fundamental issue is that the solution to an ordinary differential equation is determined by its initial condition, and there is no mechanism for adjusting the trajectory based on subsequent observations. Here, we demonstrate how this may be resolved through the well-understood mathematics of \emph{controlled differential equations}. The resulting \emph{neural controlled differential equation} model is directly applicable to the general setting of partially-observed irregularly-sampled multivariate time series, and (unlike previous work on this problem) it may utilise memory-efficient adjoint-based backpropagation even across observations. We demonstrate that our model achieves state-of-the-art performance against similar (ODE or RNN based) models in empirical studies on a range of datasets. Finally we provide theoretical results demonstrating universal approximation, and that our model subsumes alternative ODE models.

1 Introduction

The paper addresses the failure of standard sequence models and Neural ODEs on irregularly sampled or partially observed time series by introducing Neural CDEs. It establishes Neural CDEs as continuous-time RNN analogues with efficient training, strong empirical performance, and theoretical guarantees.

  • Irregular sampling and partial observation can break the usual RNN discretisation, often requiring binning or imputation.
  • Neural ODE trajectories are fixed by their initial condition, leaving no direct mechanism to incorporate observations that arrive later.
  • Controlled differential equations resolve this limitation by extending Neural ODEs into the neural controlled differential equation model for incoming data.
  • Neural CDEs process irregularly sampled and partially observed data and support memory-efficient adjoint-based backpropagation even across observations.
  • The model achieves state-of-the-art performance against similar ODE- or RNN-based models and is shown to be a universal approximator that subsumes related ODE models.

2 Background

Controlled differential equations define a continuous hidden path driven by an observed bounded-variation path, with dynamics specified through a Riemann–Stieltjes integral. Under global Lipschitz conditions, the resulting equation has a unique globally existing solution.

  • Controlled differential equations: A CDE defines a continuous path z from an initial state ζ and a continuous function f driven by a bounded-variation path X.X may be Lipschitz, and f maps R^w to R^(w×v).
  • Controlled differential equations: The driving term f(z_s)dX_s denotes matrix-vector multiplication inside a Riemann–Stieltjes integral.The subscript notation indicates function evaluation.
  • Controlled differential equations: Equation (2) has global existence and uniqueness when f satisfies global Lipschitz conditions, and is called a CDE controlled or driven by X.This terminology is distinct from the separate field of control theory.

3 Method

Neural CDEs interpolate irregularly sampled observations into a continuous path that drives hidden-state dynamics, allowing the model to adapt as new data arrives. The formulation supports universal approximation, memory-efficient adjoint training, partially observed data, and efficient batching.

  • Model construction: A natural cubic spline X through timestamped observations provides the continuous driving path for the Neural CDE.The spline has knots at all observation times and approximates the underlying process.
  • Model construction: Unlike Neural ODEs driven only by the identity function, Neural CDE dynamics change with the data path X as observations arrive.The initial hidden state is ζθ(x0, t0), and predictions may use the evolving or terminal hidden state.
  • Expressivity: A linear map applied to a Neural CDE’s terminal value is a universal approximator from sequences in Rv to R.The result follows from CDEs’ ability to approximate bases of functions on path space.
  • Comparison with alternatives: Neural CDEs can represent functions that alternative equations with nonlinear direct dependence on Xs cannot represent exactly, and they substantially outperform GRU-ODE experimentally.The stated explanation is that Neural CDEs can easily represent the identity function between paths.
  • Training and memory: O(L + H) memory is sufficient for training because continuous data incorporation permits adjoint backpropagation without interrupting the differential equation.The underlying data contributes O(L) memory, while adjoint computation contributes O(H).
  • Irregular and partial observations: Partially observed data are handled by independently interpolating each channel, while concatenated observation indices provide a non-learnt intensity signal that influences the vector field.Each channel can also receive its own observational-intensity channel that increments whenever that channel is observed.

4 Experiments

Experiments benchmark Neural CDEs against ODE- and RNN-based models on irregularly sampled, partially observed, and regularly spaced time series. Neural CDEs achieve the strongest overall performance, remain memory-efficient, and are robust as observations are removed or observational intensity is included.

  • Experimental setup: The benchmark compares Neural CDEs with GRU-∆t, GRU-D, GRU-ODE, and ODE-RNN models representing ODE- and RNN-based approaches.GRU-∆t is the straightforward baseline; GRU-D and ODE-RNN are state-of-the-art models for these problems, while GRU-ODE is thematically similar to Neural CDEs.
  • Experimental setup: Each model is run five times, with mean and standard deviation reported for test metrics, using comparable parameter counts and hyperparameters tuned through the baseline ODE-RNN.Hyperparameters are selected by grid search for ODE-RNN and then adjusted slightly across models to maintain comparable numbers of parameters.
  • Irregular time series: On CharacterTrajectories, Neural CDEs outperform every considered model, use an order of magnitude less memory, and maintain roughly constant performance as 30%, 50%, or 70% of observations are dropped.The dataset contains 2858 irregularly sampled character trajectories after uniformly random, channel-wide observation removal.
  • Partially observed time series: On PhysioNet sepsis prediction, Neural CDEs achieve the best overall AUC with observational intensity and remain the most memory-efficient model.The dataset is highly imbalanced, with a 5% positive rate, and only 10.3% of values observed.
  • Partially observed time series: Without observational intensity, every model performs substantially worse, with the decline particularly dramatic for Neural CDEs, although competing models retain access to observation-time differences.For Neural CDEs and GRU-ODE, observational intensity is continuous and channel-specific; other models receive an observed/not-observed mask, while also receiving ∆t.
  • Regularly spaced time series: On regularly spaced Speech Commands, Neural CDEs obtain the highest performance with little memory, train consistently, and are robust to optimization hyperparameter choices.The best GRU-∆t, GRU-D, and ODE-RNN models match Neural CDE performance, whereas GRU-ODE consistently fails and other benchmarks show large variance from training failures.

5 Related work

Prior work has connected Neural ODEs with recurrent updates, stochastic dynamics, rough-path methods, and signature-based time-series analysis. Other studies improve Neural ODE training, parameter variation, and regularization through diverse techniques.

  • Neural ODE variants: Neural ODEs have been interrupted with recurrent-cell updates at observations, while exponential decays provide a special case and stochastic events offer a related variant.These approaches are cited as, [9], and.
  • Neural SDEs: Neural SDEs have been used to model time-series distributions, regularize stochastic dynamics, and promote simpler trajectories with random vector fields.References [36] [38] focus on generative models, [39] [40] on regularization, and [41] on random vector fields.
  • Neural SDEs: Several methods address adjoint-backpropagation challenges for Neural SDEs, including rough-analysis treatments that approximate Brownian noise with trainable random Neural ODEs.The rough-analysis approach may also avoid the poorer convergence rates of SDE solvers relative to ODE solvers.
  • Rough analysis and CDE applications: Rough analysis and CDE theory have supported time-series applications of signature and logsignature transforms, kernels, Gaussian processes, and machine-learning software.The cited work spans [23, 45, 46, 47, 48, 49, ?, ?] through [53].
  • Neural ODE improvements: Research on Neural ODEs has explored faster training, energy-based frameworks, adjoint-backpropagation pitfalls, time-varying parameters, and regularization.The cited studies include [55], [56], [29], [57], and [55] [58].

6 Discussion

The discussion highlights two subtle construction choices—explicitly including time and conditioning the initial hidden state—as well as implementation benefits and limitations. Key limitations concern computation, parameter counts, noncausal interpolation, uncertainty, numerical schemes, and the scope of experiments.

  • Construction choices: Time must be included explicitly as a channel because CDEs are invariant to the speed at which X is traversed.The construction uses X_ti = (x_i, t_i).
  • Construction choices: The initial hidden state z_t0 should depend on X_t0; otherwise, the model depends on X only through dX/dt and is translationally invariant.An alternative is adding a translation-sensitive channel such as t_i x_0.
  • Computational considerations: Neural CDEs were typically slightly faster to compute than the ODE-RNN model of because solvers can step across observations.ODE-RNN solvers must interrupt at each observation.
  • Computational considerations: Neural CDEs were still roughly fives times slower than RNN models, which the authors attribute largely to implementation choices.The implementation uses Python, double precision, and variable step size solvers by default.
  • Model limitations: The final affine transformation can require O(ωvw) parameters, forcing small w and ω to match parameter counts across models.A rank-one outer-product parameterization reduces this to O(ω(v + w)) but hindered classification performance.
  • Model limitations: As presented, Neural CDEs provide no uncertainty measure, while alternative numerical CDE methods and more sophisticated vector fields may improve performance.The paper also reports experiments only on classification problems and expects broader applicability.
  • Model limitations: Natural cubic splines make X noncausal, preventing real-time application because X_t depends on x_i for t < t_i.The authors identify resolving this issue as planned follow-up work.

7 Conclusion

The paper introduces Neural CDEs, a continuous-time analogue of RNNs for irregularly sampled and partially observed multivariate time series. The model also achieves state-of-the-art performance and supports memory-efficient adjoint-based backpropagation across observations.

  • 7 Conclusion: Neural CDEs are a new class of continuous-time time series models and the continuous-time analogue of RNNs.The paper contrasts Neural CDEs with Neural ODEs, which are described as the continuous analogue of ResNets.
  • 7 Conclusion: Neural CDEs operate directly on irregularly sampled and partially observed multivariate time series.This capability is identified as one of the model’s three key advantages.
  • 7 Conclusion: Neural CDEs demonstrate state-of-the-art performance and enable memory-efficient adjoint-based backpropagation even across observations.These are the remaining two key advantages listed in the conclusion.

Broader Impact · A Other schemes for constructing the path X · A.1 Differentiating with respect to the time points

The paper frames Neural CDEs as a tool with anticipated positive impact, particularly for electronic health records, while documenting path regularity requirements and a caveat for differentiating with respect to initial time. Under adjoint training, this differentiation requires time derivatives of the vector field and, in practice, a twice-differentiable path.

  • Broader Impact: The authors anticipate positive impact from applying the tool to irregularly sampled electronic health records and no specific negative impacts.Electronic health records are identified as an important example of irregularly sampled time-stamped data.
  • A Other schemes for constructing the path X: For model evaluation, the constructed path X must be continuous and piecewise differentiable.This regularity condition is stated for evaluating the model as discussed in Section 3.2.
  • A.1 Differentiating with respect to the time points: Differentiating with respect to the initial time t0 during adjoint-based training requires derivatives of the vector field with respect to t.This is a technical caveat specific to the combination of initial-time derivatives and adjoint training.
  • A.1 Differentiating with respect to the time points: The appendix analyzes derivatives of a scalar-valued function L of the terminal state z_tn through the Neural CDE.L may represent a loss, and the associated derivative is vector-valued with size equal to the hidden-channel dimension.
  • A.1 Differentiating with respect to the time points: The derivation applies [15, Equation 52], with the dot product denoted by ·.The cited equation is applied to the Neural CDE case to obtain the relevant differentiation formula.
  • A.1 Differentiating with respect to the time points: Although equation (6) can in principle accommodate measure-valued d2X/ds2, practical implementations generally require X to be twice differentiable.Most code is described as handling only classical derivatives, motivating the stronger practical regularity requirement when derivatives with respect to t0 are desired.

A.2 Adaptive step size solvers · A.3 Natural cubic splines

Adaptive solvers can become inefficient when the interpolated path has discontinuous second derivatives, while natural cubic splines provide the minimum smoothness needed for acceptable operation. Beyond smoothness, alternative interpolation, approximation, and curve-fitting methods remain viable.

  • A.2 Adaptive step size solvers: A twice-differentiable path with discontinuous second derivative, such as a quadratic spline interpolation, creates this solver caveat.The discontinuity in the second derivative is the source of the adaptive solver difficulty.
  • A.2 Adaptive step size solvers: Adaptive solvers took a long time on the backward pass because they slowed to resolve each jump in d2X/ds2 and sped up between intervals.The issue arises when solving equation (6) with an adaptive step-size solver.
  • A.3 Natural cubic splines: Natural cubic splines ensure that X is twice continuously differentiable, ameliorating the solver issue and allowing adaptive step-size solvers to operate acceptably.Their smoothness directly addresses the jumps in d2X/ds2 encountered during adaptive solving.
  • A.3 Natural cubic splines: Cubic splines provide essentially the minimum regularity required for the paper’s techniques to work out of the box in all cases.The regularity requirement is tied to acceptable adaptive solver behavior.
  • A.3 Natural cubic splines: Natural cubic splines are not otherwise special beyond their smoothness.The paper presents smoothness, rather than another distinctive property, as the reason for selecting them.
  • A.3 Natural cubic splines: Gaussian processes and kernel methods are possible alternatives, while noisy data may instead use approximation or curve-fitting schemes rather than interpolation.The passage permits both alternative path-construction methods and non-interpolating schemes.

B Universal Approximation

This section establishes universal approximation for controlled differential equations by reducing them to the signature transform, then extends the result to irregular time series represented by natural cubic splines. Neural CDEs are dense in continuous functions on compact time-series spaces under the natural cubic spline topology.

  • Signature approximation: The signature transform provides a universal nonlinearity: linear functionals of truncated signatures are dense in continuous functions on compact path spaces.The proof reduces neural CDE approximation to the signature transform, which acts as a basis for functions on compact path space.
  • CDE approximation: CDE input-output maps are dense in C(K; R^u) for compact sets K of bounded-variation controlling paths.The approximating family uses a CDE vector field, terminal linear functional, and initial condition depending on the initial control value.
  • Spline approximation: Uniformly bounded C^2 path families with uniformly bounded first and second derivatives are relatively compact in the bounded-variation path space.This compactness result supports transferring path-space approximation to families of spline-controlled inputs.
  • Spline approximation: Neural CDE input-output maps are dense in C(K; R^u) under the natural cubic spline topology on time series.The theorem extends universal approximation from continuous bounded-variation paths to the original irregularly sampled data represented by natural cubic splines.

C Comparison to alternative ODE models

The Neural CDE model strictly subsumes alternative ODE models whose vector fields depend directly and nonlinearly on the control. This inclusion is algebraic: the CDE can represent every alternative-ODE trajectory and additional trajectories unavailable to the alternative formulation.

  • Strict inclusion: Theorem C.1 establishes that the alternative ODE model class is a strict subset of the Neural CDE class, Y ⊊ Z.The result compares projected Neural CDE solutions with alternative ODE solutions under the stated dimensional and regularity assumptions.
  • Model construction: The CDE achieves implicit nonlinear dependence on X by recording X in v+1 hidden channels, although it uses a larger state space of w rather than w−v−1.Time augmentation is included to avoid restrictions from CDE reparameterisation invariance, though it is not required for strictness.
  • Scope: The theorem is an algebraic comparison and does not make analytic claims such as universal approximation.In practice, the functions defining either model may correspond to trained neural networks.
  • Strict inclusion: Some Neural CDE trajectories cannot be represented by the alternative ODE because their derivatives depend on information beyond t, X_t, and X_τ.The contradiction arises because the alternative vector field would need to determine dX^1/dt(t) from insufficient variables.
  • Strict inclusion: Every alternative ODE trajectory can be realised as the projection of a Neural CDE trajectory, so Y ⊆ Z.The construction selects an augmented initial state and a CDE vector field whose projected solution satisfies the same differential equation; uniqueness then gives equality of trajectories.

D Experimental details … D.4 Speech Commands

The experiments used standardized data and shared optimization and solver conventions, while each dataset had tailored preprocessing, splits, architectures, and training schedules. Hyperparameters were selected around the ODE-RNN baseline and adjusted across models to maintain comparable parameter counts.

  • D.1 General notes: All datasets were normalized so that every channel had mean zero and variance one, with binary or multiclass tasks using the corresponding cross-entropy objective.Binary classification used binary cross-entropy after a sigmoid; multiclass classification used cross-entropy after a softmax.
  • D.1 General notes: Neural CDE and ODE-RNN integrands were feedforward networks with a final linear output layer, using ReLU for Neural CDEs and tanh for ODE-RNNs.ODE components used the fourth-order Runge-Kutta 3/8-rule solver with step size equal to the minimum adjacent observation-time difference.
  • D.2 CharacterTrajectories: CharacterTrajectories used a 70%/15%/15% train/validation/test split and initialized Neural CDE hidden states with a learned linear map from the first observation.Selected models used 32 hidden channels for Neural CDE and ODE-RNN, 47 for the GRU baselines, and three width-32 hidden layers for Neural CDE and ODE-RNN vector fields.
  • D.3 PhysioNet sepsis prediction: PhysioNet sepsis prediction used a 70%/15%/15% split and encoded static features through a width-256 single-hidden-layer ReLU network controlling each model’s initial condition.Missing values for several baseline models were filled with natural cubic splines, while the observational mask was also passed for observational-intensity inputs.
  • D.3 PhysioNet sepsis prediction: PhysioNet hyperparameters were optimized for ODE-RNN and then adjusted to equalize parameter counts, selecting 128 hidden channels and a four-layer width-128 vector field for ODE-RNN.The Neural CDE configuration was reduced to 49 hidden channels and width 49, while GRU baselines increased to 187 hidden channels.
  • D.4 Speech Commands: Speech Commands training used batch size 1024 and learning rate 0.0016, with a 200-epoch maximum and a final-layer learning rate of 0.16.The learning rate was divided by 10 after 10 stagnant training-loss epochs, and training stopped after 100 stagnant training-loss or validation-accuracy epochs; selected hidden channels were 128 for ODE-RNN, 90 for Neural CDE, and 160 for GRU baselines.
  • D.4 Speech Commands: Speech Commands transformed each length-16000 univariate series into a length-161 sequence of 20 logarithmically scaled MFCCs using a Hann window of length 200 and hop length 100.The transform used 200 frequency bins and 128 mel filterbanks.
Loading 2005.08926v2…