PaperlayerView exact revision on arXiv

Source-linked AI summary

Neural Rough Differential Equations for Long Time Series

James Morrill, Cristopher Salvi, Patrick Kidger, James Foster, Terry Lyons

arXiv:2009.08295v4cs.LGcs.AImath.DSstat.ML

TL;DR

Long time series make Neural CDE training slower and less effective because existing approaches require many forward evaluations. The paper introduces Neural RDEs, which use interval log-signatures and the log-ODE method, and reports speed, performance, and memory benefits on sequences up to 17 000 observations. These benefits come with additional hyperparameters and reduced feasibility as input-channel count grows.

  • Problem

    Neural CDEs begin to break down on long time series, with worsening loss or accuracy and prohibitive training time from many forward operations.

  • Method

    Neural RDEs use local log-signatures to update Neural CDE hidden states over larger intervals, while retaining continuous-time adjoint backpropagation.

  • Results

    Neural RDEs show substantial speed, model-performance, and memory improvements on real-world problems of length up to 17 000.

  • Takeaways & Limitations

    Reducing the effective control-path length makes Neural RDEs particularly beneficial for long time series while preserving continuous dynamics and memory-efficient backpropagation.

  • Takeaways & Limitations

    The approach introduces truncation-depth and step-size hyperparameters, and becomes less feasible with many input channels because log-signature dimension grows exponentially in channel count.

Abstract

from arXiv · show

Neural controlled differential equations (CDEs) are the continuous-time analogue of recurrent neural networks, as Neural ODEs are to residual networks, and offer a memory-efficient continuous-time way to model functions of potentially irregular time series. Existing methods for computing the forward pass of a Neural CDE involve embedding the incoming time series into path space, often via interpolation, and using evaluations of this path to drive the hidden state. Here, we use rough path theory to extend this formulation. Instead of directly embedding into path space, we instead represent the input signal over small time intervals through its \textit{log-signature}, which are statistics describing how the signal drives a CDE. This is the approach for solving \textit{rough differential equations} (RDEs), and correspondingly we describe our main contribution as the introduction of Neural RDEs. This extension has a purpose: by generalising the Neural CDE approach to a broader class of driving signals, we demonstrate particular advantages for tackling long time series. In this regime, we demonstrate efficacy on problems of length up to 17k observations and observe significant training speed-ups, improvements in model performance, and reduced memory requirements compared to existing approaches.

1. Introduction

Neural CDEs model time-varying data continuously, but their forward computation becomes costly and degrades on long sequences. Neural RDEs use interval log-signatures to reduce the effective sequence length while retaining continuous-time dynamics and adjoint backpropagation.

  • Neural CDEs: Neural CDEs are continuous-time neural models whose vector field depends on time-varying observations, driving an evolving hidden state.They are presented as continuous-time analogues of recurrent neural networks and can be solved using Neural ODE tools when the control is differentiable.
  • Motivation: For long time series, Neural CDEs suffer worsening loss or accuracy and prohibitive training time from many forward operations.The paper relates this behavior to the corresponding difficulty experienced by recurrent neural networks.
  • Neural RDEs: Neural RDEs replace pointwise control evaluations with log-signature summaries computed over short intervals to update the hidden state.The log-ODE method applies these updates over intervals larger than individual sampling steps and does not require differentiable control paths.
  • Neural RDEs: Neural RDEs retain continuous dynamics and memory-efficient adjoint backpropagation while reducing the effective length of the control path.The approach is intended to address long-series computational and memory pressure without abandoning the Neural CDE framework.
  • Results: Experiments on real-world problems up to 17 000 observations report improvements in model performance, speed, and memory usage.The reported gains include up to 17% on some classification tasks, roughly 10× speed, and roughly 100× lower memory usage than models not using the adjoint method.

2. Theory

Signature transforms summarize how paths interact with dynamical systems, while log-signatures remove redundancy from these statistics. The log-ODE method uses local log-signatures to approximate CDE solutions with ODEs over successive intervals.

  • Signatures: The signature transform maps a path to real-valued statistics describing how its data interacts with controlled dynamical systems.Under mild conditions, the infinite signature can completely determine a path up to translation when time is included as a channel.
  • Signatures: A depth-N signature consists of iterated integrals whose values are independent of the chosen terminal time and partition points.The paper presents the signature for continuous, piecewise differentiable paths and notes that it is commonly applied to linearly interpolated time series.
  • Log-signatures: The log-signature compresses the signature by removing redundant terms, yielding a fixed-basis representation in R^β(v,N).Its dimension β(v,N) depends on the number of channels v and truncation depth N.
  • Geometric intuition: At depth 1, log-signature terms encode channel changes; at depth 2, they encode signed path area relative to the endpoint chord.Higher depths represent higher-order integrals and iterated areas.
  • Log-ODE method: The log-ODE method approximates a CDE locally by solving an ODE driven by the interval’s log-signature, then repeats this over a partition.The approximation arises from the signature-based Taylor expansion of the CDE and is termed an RDE formulation.

3. Method

Neural RDEs replace pointwise control-path evaluations with log-signatures over fewer, larger intervals, allowing standard ODE solvers to update the hidden state efficiently. This formulation generalises Neural CDEs and is particularly useful for long time series through compression, larger solver steps, and memory-efficient adjoint backpropagation.

  • Neural RDE construction: Neural RDEs divide the interpolated input path into intervals and represent each interval with a depth-N log-signature summarising how it drives the CDE.The interval endpoints r_i are typically equally spaced, with m chosen much smaller than the original sequence length n.
  • Neural RDE construction: The log-signature replaces the Neural CDE driving term with a neural-network matrix multiplied by the interval log-signature, yielding an ODE solvable by standard ODE solvers.The network maps the hidden state into the dimensions required by the log-signature, and the resulting equation can be solved directly as a neural ODE.
  • Neural RDE construction: Neural RDEs recover the original Neural CDE result under linear interpolation, while extending the formulation to the log-signature-based rough differential equation approach.At depth 1, the log-signature is the path increment, giving the same result as the original method with linear interpolation.
  • Efficiency for long series: A sequence of length m much smaller than n varies more slowly, so the driven differential equation permits larger integration steps and produces typical speed-ups of about 10×.The log-signature path is higher-dimensional but requires fewer solver steps because it is more slowly varying.
  • Efficiency for long series: The adjoint method reduces backpropagation memory from O(HT) to O(H + T), with one experiment decreasing usage from 3.6GB to 47MB.Both Neural CDEs and Neural RDEs support continuous-time adjoint backpropagation, avoiding the memory demands of backpropagation-through-time.
  • Implementation and scope: Log-signatures can be computed once during preprocessing or online, and the method can be implemented with existing log-signature, ODE, or Neural CDE tools.The authors use Signatory for log-signatures and note that the log-signature path can be handled with existing Neural CDE tooling.
  • Implementation and scope: The method offers limited benefit on relatively short time series because the original Neural CDE formulation already works well enough for substantial speed or accuracy gains.Its principal advantages are therefore directed toward the long-time-series regime, where the existing approach struggles as sequence length grows.
  • Efficiency for long series: Neural RDE solver steps can be decoupled from data sampling rate, while step size and log-signature depth remain modelling hyperparameters trading speed against information.Increasing step size accelerates training but is less informative; increasing depth slows training while incorporating more local-interval information.

4. Experiments

Experiments on long, regularly sampled time series compare Neural RDEs with Neural CDEs and ODE-RNNs across step sizes and depths. Neural RDEs generally improve accuracy or test loss while reducing training time, whereas ODE-RNNs incur higher memory costs and poorer performance.

  • Experimental setup: Experiments cover four long real-world datasets, including EigenWorms sequences of length 17 984 and three vital-sign prediction tasks with sequences of length 4 000.The vital-sign tasks predict respiratory rate, heart rate, or oxygen saturation from PPG and ECG data using L2 loss.
  • EigenWorms classification: Neural RDEs reduce EigenWorms training time from roughly one day for step-1 Neural CDEs to roughly two hours, while improving accuracy by up to 17%.The comparison uses test-set accuracy, mean memory usage, and training time over repeated runs.
  • EigenWorms classification: Naïve Neural CDE subsampling achieves speed-ups without performance improvements, while folded ODE-RNNs attain the worst scores and impose a significantly higher memory burden.The ODE-RNN benchmark folds adjacent time points into the feature dimension to address long sequences.
  • Vital-sign estimation: Depth-3 Neural RDEs are top performers for every vital-sign task at every step size, reducing test loss by 30–59% versus Neural CDEs.They achieve this with roughly an order of magnitude less training time.
  • Vital-sign estimation: Neural RDE performance improves as step size increases, unlike Neural CDE performance, which degrades; the authors attribute the gain to learning long-term dependencies through reduced sequence length.ODE-RNN performance improves when sequences are shortened but remains substantially worse than Neural RDE performance and uses significantly more memory.

5. Limitations

The method introduces two tunable hyperparameters and is most feasible when the input has few channels. Higher log-signature depth can increase channel growth and saturate available parallelism.

  • Hyperparameter tuning: Two new hyperparameters—truncation depth and step size—have substantial effects on training time and memory usage and must be tuned.These choices add tuning requirements beyond the Neural CDE formulation.
  • Input-channel scaling: The log-ODE method is most feasible with few input channels because the number of log-signature channels grows exponentially with truncation depth.For larger depths, available parallelism may become saturated.

6. Related Work

Related work addresses long-sequence modelling through architectural changes, subsampling, and continuous-time formulations. The paper positions its log-ODE approach as an implementation that preserves differential-equation structure while combining log-signatures with Neural CDEs.

  • Long time series: Transformers can improve long-term dependency modelling but typically have O(L^2) complexity, motivating more efficient long-sequence approaches.Li et al. (2019) reduce this to O(L(log L)^2).
  • Long time series: Unitary and orthogonal RNNs mitigate vanishing or exploding gradients, whereas dilated and Skip-RNN models target training speed and long-term dependencies.Unitary or orthogonal RNNs require a matrix inversion at each training step.
  • Subsampling and signatures: Hierarchical subsampling processes windows with an RNN/RNN pair, while Liao et al. use a log-signature/RNN pair.The paper describes its log-ODE method as analogous to a log-signature/NCDE pair.
  • Experimental comparison: Table 2 evaluates three BIDMC vital-sign prediction tasks using test L2 loss, memory usage, training time, step sizes, and Neural RDE depths.The NCDE is treated as a depth-1 Neural RDE; unavailable GPU-memory runs are marked with ‘–’.
  • Experimental comparison: Figure 5 visualises normalised losses across BIDMC datasets, step sizes, and depths, exposing a depth/step tradeoff for long time series.The lowest MSE occurs with step > 1 and depth > 1, while very long steps worsen performance.
  • Positioning the contribution: The paper moves beyond Liao et al.’s preprocessing choice to implement the log-ODE method while preserving differential-equation structure.It also exploits the relationship between log-signatures and the controlled differential equation they drive.
  • Continuous-time recurrent models: Continuous-time modifications of GRUs and LSTMs improve learning of long-term dependencies, while ODE-based work targets long-term memory capacity.The paper identifies hybridising these techniques with its differential-equation approach as future work.

7. Conclusion

The conclusion presents Neural RDEs as a continuous-time approach that updates hidden states using interval summaries rather than point evaluations. For long time series, this reduces the effective control-path length while retaining continuous dynamics and adjoint backpropagation.

  • 7. Conclusion: Neural RDEs extend Neural CDEs by driving hidden states with interval summarisations of the underlying time series or control path.They retain continuous dynamics and can still be solved with ODE methods.
  • 7. Conclusion: Neural RDEs reduce the effective control-path length and show training-speed, performance, and memory benefits on problems up to 17 000 observations.The conclusion reports these benefits specifically in the long-time-series regime.

Supplementary material

The supplementary material expands the mathematical and experimental documentation. It introduces log-ODE solution methods, records implementation details, and provides a full breakdown of experimental results.

  • Supplementary material: Sections A and B provide a more thorough introduction to solving controlled differential equations via the log-ODE method.
  • Supplementary material: Section C documents network structure, computing infrastructure, and the hyperparameter-selection approach.
  • Supplementary material: Section D gives a full breakdown of every experimental result.

A. An introduction to the log-ODE method for controlled differential equations

The log-ODE method approximates controlled differential equations by combining path summaries with derivatives of the vector field, typically applied over successive local intervals. Its Taylor and log-ODE variants use truncated signatures or log-signatures, with assumptions ensuring the resulting methods are well defined.

  • Log-ODE construction: The log-ODE method approximates a controlled differential equation using an ordinary differential equation.It is applied locally by splitting the interval and solving across successive subintervals.
  • Assumptions: The vector field is assumed either bounded with bounded derivatives or linear, ensuring the Taylor and log-ODE methods are well defined.These assumptions make the constructed vector field globally bounded and Lipschitz continuous or linear.
  • Path signatures: The truncated signature is built in the truncated tensor algebra, whose levels contain tensor powers through depth N.The algebra includes R, R^d, (R^d)^⊗2, through (R^d)^⊗N.
  • Path signatures: A path signature collects iterated integrals describing how a path affects a controlled differential equation, while the log-signature compresses the same information.The log-signature removes algebraic redundancies from the signature.
  • Taylor and log-ODE methods: The Taylor method uses a truncated expansion involving the path signature and recursively defined derivatives of the vector field.The derivatives are Fréchet derivatives of recursively constructed vector-field derivatives.
  • Taylor and log-ODE methods: The log-ODE method applies the Taylor-derived vector field to a truncated log-signature to define an ODE whose solution approximates the CDE.The construction uses the function bf(z) := Taylor(z, f, ·).
  • Examples: When the truncation depth is one, the increment-only log-ODE method is equivalent to driving the original CDE with a piecewise linear control-path approximation.This is a classical approach for stochastic differential equations and a Wong–Zakai approximation.
  • Examples: For the inhomogeneous geometric Brownian motion example, the log-ODE method uses the control path X = {(t, W_t)} and truncation depth N = 3.The step size is denoted h := t − s.

B. Convergence of the log-ODE method for rough differential equations

This section defines the log-ODE approximation for rough differential equations and establishes its well-posedness and convergence under stated regularity assumptions. It also explains how truncation depth, step size, and control-path roughness affect approximation quality.

  • Well-posedness: Under Lip(γ) regularity with γ > p, the RDE has a unique solution in the Lyon’s sense.The setup assumes a geometric p-rough path and a vector field with bounded, Hölder-regular derivatives.
  • Log-ODE construction: The log-ODE method computes a truncated log-signature over each interval, constructs a vector field from it, and solves an associated ODE.The solution at u = 1 approximates the RDE solution over the interval.
  • Error behavior: Higher convergence rates follow from increasing log-signature truncation depth, while greater control-path roughness increases the error estimate.The section recommends choosing step size and truncation depth appropriately.
  • Approximation theory: Log-ODEs approximate controlled differential equations, while linear RDEs require separate boundedness arguments because their vector fields may be unbounded.For linear vector fields, the log-ODE vector field is linear and its solution is given by a matrix exponential.
  • Global approximation: A global numerical solution is obtained by applying the interval-wise log-ODE construction across a partition whose local p-variation is sufficiently small.The resulting global error is controlled by a constant depending on p, γ, and the vector-field regularity.

C. Experimental details

The experiments use fixed preprocessing, optimization, architecture, solver, and hardware choices, with hyperparameters selected through validation-based searches. The architecture uses bounded final nonlinearities to help control long-sequence dynamics.

  • Data preparation: Each dataset is split into training, validation, and testing sets with relative sizes 70%/15%/15%.Training-set statistics are used to normalize all three splits to zero mean and unit variance.
  • Hyperparameter selection: Neural CDE hyperparameters are selected by grid search at 500 sequence steps, and Neural RDE models reuse the Neural CDE hyperparameters.The search balances sequence length and training time.
  • Architecture: The hidden-state update network is a multilayer perceptron with ReLU nonlinearities except for a final tanh nonlinearity.The stated goal of the final tanh is to help prevent term blow-up over long sequences.
  • Implementation caveat: The final hidden-layer tanh placement differs slightly from the original Neural CDE model, although both choices constrain the hidden state's rate of change.The authors report no reason to prefer one placement over the other.
  • Training and infrastructure: The implementation uses the Adam optimizer, adaptive learning-rate reductions, early stopping, and the rk4 solver from torchdiffeq version 0.0.1.Experiments ran on RTX 2080 Ti and Quadro GP100 GPUs.

D. Experimental Results

This section presents full experimental breakdowns for EigenWorms and BIDMC, including accuracy or prediction loss, memory usage, and training time across model configurations. The tables also document hyperparameter searches and cases where models could not run or train effectively.

  • Result breakdown: Tables 7 and 8 report the complete EigenWorms and BIDMC experimental results, respectively.EigenWorms results include accuracy, memory usage, and training time; BIDMC results include L2 losses, time, and memory.
  • Hyperparameter results: The hyperparameter tables document selected settings for EigenWorms, BIDMC, and ODE-RNN variants.The BIDMC ODE-RNN table notes ineffective training for HR and SpO2, where validation losses remained equal to two decimal places.
  • EigenWorms: EigenWorms results compare test accuracy, memory usage, and training time across depths 1–3 and selected step sizes.Values are means and standard deviations over three repeats, with bold entries identifying the top performer for each step size.
  • BIDMC: BIDMC results compare mean L2 test losses for RR, HR, and SpO2 prediction tasks across three repeats.Memory is averaged across tasks, and errors indicate models that could not run within GPU memory.
Loading 2009.08295v4…