On Neural Differential Equations

Patrick Kidger

Source-linked AI summary

On Neural Differential Equations

Patrick Kidger

arXiv:2202.02435v1Feb 4, 2022cs.LGmath.CAmath.DSmath.NAstat.ML

TL;DR

The rapidly developing field of neural differential equations had no survey on the topic to the authors’ knowledge. This thesis presents an accessible survey spanning applications in mathematical modelling and machine learning, intended as a useful reference for interested readers.

Problem
To the authors’ knowledge, the rapidly developing field of neural differential equations lacked a survey, despite spanning mathematical modelling and machine-learning applications.
Source: §1.1 · p. 15
Method
The thesis combines original research with an accessible survey of neural differential equations.
Source: §1.1 · p. 15
Results
The thesis covers a wide variety of applications in classical mathematical modelling and typical machine-learning problems.
Source: §1.1 · p. 15
Takeaways & Limitations
The authors intend the thesis to be useful to readers interested in neural differential equations.
Source: §1.1 · p. 15
For image classification, continuous-time models usually offer little benefit, and explicitly discretised residual networks are easier to work with.
Source: §2.2.1 · p. 24

Abstract

from arXiv · show

The conjoining of dynamical systems and deep learning has become a topic of great interest. In particular, neural differential equations (NDEs) demonstrate that neural networks and differential equation are two sides of the same coin. Traditional parameterised differential equations are a special case. Many popular neural network architectures, such as residual networks and recurrent networks, are discretisations. NDEs are suitable for tackling generative problems, dynamical systems, and time series (particularly in physics, finance, ...) and are thus of interest to both modern machine learning and traditional mathematical modelling. NDEs offer high-capacity function approximation, strong priors on model space, the ability to handle irregular data, memory efficiency, and a wealth of available theory on both sides. This doctoral thesis provides an in-depth survey of the field. Topics include: neural ordinary differential equations (e.g. for hybrid neural/mechanistic modelling of physical systems); neural controlled differential equations (e.g. for learning functions of irregular time series); and neural stochastic differential equations (e.g. to produce generative models capable of representing complex stochastic dynamics, or sampling from complex high-dimensional distributions). Further topics include: numerical methods for NDEs (e.g. reversible differential equations solvers, backpropagation through differential equations, Brownian reconstruction); symbolic regression for dynamical systems (e.g. via regularised evolution); and deep implicit models (e.g. deep equilibrium models, differentiable optimisation). We anticipate this thesis will be of interest to anyone interested in the marriage of deep learning with dynamical systems, and hope it will provide a useful reference for the current state of the art.

Originality

The thesis distinguishes the author’s original work from prior or concurrent material included for survey, and documents individual contributions to its papers, open-source software, and previously unpublished topics.

Statement

The thesis states that its writing is original and that included material is either original work, with or without collaborators, or prior or concurrent work included for survey.This frames the thesis’s originality statement without claiming that every included work was solely authored by the thesis writer.Source: Statement · p. 10

Open source software

A substantial part of the PhD work was developing open-source software to make neural differential equations more accessible, including differential-equation solvers for JAX and PyTorch.The listed tools include ordinary, controlled, and stochastic differential-equation solvers for JAX, plus ODE, CDE, and SDE solvers for PyTorch.Source: Open source software · p. 11, Diffrax · p. 11

Breakdown of contributions

For the irregular-time-series neural CDE paper, the author says they did the entirety of the work; two coauthors had concurrently pursued similar ideas and were included as a courtesy.The thesis attributes the paper to Patrick Kidger, James Morrill, James Foster, and Terry Lyons.Source: Breakdown of contributions · p. 11
For the ODE-adjoints paper, the author contributed the idea, theory, library implementation, and neural CDE and Hamiltonian experiments; for the rough-DE paper, they identified its long-time-series application.The ODE-adjoints paper’s continuous normalising-flow experiments were by Ricky T. Q. Chen, with text written jointly; the rough-DE paper’s log-ODE idea was Cristopher Salvi’s, its implementation James Morrill’s, and its theory assisted by James Foster, with text jointly written by the author and Morrill.Source: Breakdown of contributions · p. 11
For the infinite-dimensional GAN paper, the author contributed the basic idea, basic theory, and experimental code, then jointly developed torchsde with Xuechen Li for the paper.James Foster contributed SDE numerics knowledge; the fuller idea and text were developed jointly by the three authors. For the gradients paper, the author contributed the Brownian Interval and gradient-penalty-free training ideas and theory, and wrote all code; James Foster devised the reversible Heun method.Source: Breakdown of contributions · pp. 11–12
For the online-prediction neural CDE paper, the author contributed the idea and abstract theory; James Morrill developed cubic Hermite splines with backward differences and implemented them, while Lingyi Yang assisted with datasets.Terry Lyons is identified as the author’s supervisor and included on each paper.Source: Breakdown of contributions · p. 12

Previously unpublished

The thesis includes previously unpublished material on symbolic regression, universal approximation, neural differential-equation parameterisations, and differential-equation sensitivities.The thesis describes this material as usually about half a paper in size.Source: Previously unpublished · p. 12

Introduction

The thesis presents original research alongside what its authors describe as the first survey of neural differential equations, covering applications in mathematical modelling and machine learning.

1.1.1 Getting started

The thesis assumes basic ODE and modern deep-learning knowledge, plus SDE familiarity for neural SDE material, while introducing other concepts as needed and recommending sequential reading.Numerical examples generally use toy datasets; associated code is available in Diffrax for JAX, and real-world applications are discussed in the cited original papers.Source: §1.1.1 · pp. 15–16

1.1.2 What is a neural differential equation anyway?

A neural differential equation uses a neural network to parameterize a differential equation’s vector field, with the differential equation solver embedded in a differentiable computation graph.The resulting computation can be backpropagated through and trained using stochastic gradient descent.Source: §1.1.2 · pp. 16–17

1.1.3 A familiar example

The SIR model illustrates NDEs through a theory-derived vector field whose parameters can be fitted to observed trajectories using a loss, backpropagation, and stochastic gradient descent.This follows the same procedure as a neural ODE, changing the vector-field parameterization.Source: §1.1.3 · pp. 17–18

1.1.4 Continuous-depth neural networks

Explicit Euler discretization of a neural ODE recovers a residual-network formulation [He+15]; the thesis also connects GRU/LSTM updates, score-based diffusion models, and invertible-network coupling layers to differential-equation methods.The cited connections are to discretized differential equations for GRU/LSTM updates and diffusion models, and to reversible differential-equation solvers for coupling layers [Cho+14] [Beh+19].Source: §1.1.4 · pp. 18–19

1.1.5 An important distinction

Unlike NDEs, which use neural networks to specify differential equations, physics-informed neural networks use them to approximate solutions to prespecified equations.PINNs can be effective for some nonlocal or high-dimensional PDEs where traditional solvers are computationally expensive, though traditional solvers are more efficient in most regimes [Zub+21].Source: §1.1.5 · p. 19

1.2.1 Applications

NDEs support hybrid physical, financial, and biological modelling, learning from irregular time series, and generative modelling via differential-equation-derived maps.Continuous normalizing flows may use a normal base distribution, while neural SDEs may use the law of Brownian motion; system identification and reinforcement learning are mentioned but not treated heavily.Source: §1.2.1 · p. 20

1.2.2 Advantages

NDEs combine neural networks’ high-capacity function approximation and trainability with differential equations’ model-space priors, memory efficiency, and established theory.The thesis frames this as unusually high modelling capacity relative to classical differential equations, together with a coherent account of good models relative to modern deep learning.Source: §1.2.2 · p. 20

1.3 A note on history

Although NDEs became a field after interest surged following [Che+18b], the thesis traces core ideas to earlier work on neural ODEs, controlled systems, mechanistic hybrids, and stiff dynamics.Examples include chemical-reaction dynamics learned with a neural ODE, controlled dynamical systems, mechanistic hybridization, and implicit integrators for stiff systems.Source: §1.3 · p. 21

Neural Ordinary Differential Equations

This section surveys neural ordinary differential equations as neural-network vector fields, including their use with physical structure, continuous normalising flows for complex distributions, and latent ODEs for irregularly sampled time series. It also covers numerical solution and gradient computation, existence and uniqueness under suitable Lipschitz conditions, and Hutchinson-based trace estimation.

2.1 Introduction

Neural ODEs [Che+18b] evolve an arbitrary-dimensional state through a neural-network vector field, with existence and uniqueness guaranteed when the field is continuous in time and uniformly Lipschitz in state.The vector field is typically a standard architecture such as a feedforward or convolutional network.Source: §2.1 · p. 22, §2.1.1 · p. 22

2.1.2 Evaluation and training

Training and evaluation require both numerical solutions, since analytic ones are generally unavailable, and backpropagation through the differential equation to obtain parameter gradients.Software for these tasks is described as standardised.Source: §2.1.2 · p. 23

2.2.1 Image classification

For image classification, continuous-time models are usually not advantageous: explicitly discretised neural ODEs, or residual networks, are easier to use, although ODE flows can evolve data manifolds.The image-classification construction is presented as an example rather than a recommended approach.Source: §2.2.1 · p. 24

2.2.2.1 Universal differential equations

Universal differential equations [Rac+20b] augment known physical models with neural-network correction terms to learn residual behavior and close gaps between theory and observations.This hybrid approach is suited to complex, poorly understood behavior when sufficient data show that the theoretical model falls short.Source: §2.2.2 · p. 24, §2.2.2.1 · p. 25

2.2.2.2 Hamiltonian neural networks

Hamiltonian neural networks [GDY19] encode dynamics through a learned Hamiltonian-derived vector field, while Lagrangian neural networks [Cra+20b] offer an alternative when canonical coordinates are unavailable.Hamiltonians may be unstructured networks or parameterized by kinetic and potential energy; the Lagrangian is a network of positions and velocities fitted to data.Source: §2.2.2.2 · pp. 26–27, §2.2.2.3 · pp. 27–28

2.2.3 Continuous normalising flows

A continuous normalising flow [Che+18b] [Gra+19] learns an ODE whose pushforward of a normal base distribution approximates a target distribution, and generates samples by solving forward from the base.The text illustrates the approach by fitting a flow to a greyscale image treated as an unnormalised density.Source: §2.2.3 · p. 28, Figure 2.1 · p. 29, §2.2.3.3 · p. 30

2.2.3.2 Instantaneous change of variables

Maximum-likelihood training uses the instantaneous change-of-variables relation, in which density change is determined by the divergence, or Jacobian trace, of the vector field.The method evaluates samples by solving the ODE backward in time and integrates the density-change term along the trajectory.Source: §2.2.3.2 · pp. 28–29

2.2.3.3 Example

Continuous normalising flows learn two-dimensional distributions with multiple modes and fine-scale filaments.The example transforms a normal distribution into target, cat, and butterfly distributions.Source: §2.2.3.3 · p. 31, Figure 2.2 · p. 31

2.2.3.4 Efficient estimation of the trace-Jacobian

Hutchinson’s trace estimator gives an unbiased divergence estimate in O(d) work, using a vector-Jacobian product and dot product.In practice, a single Monte Carlo sample is often held constant during integration; batch averaging during training often makes additional samples unnecessary.Source: §2.2.3.4 · p. 32

2.2.3.5 Comparison to normalising flows

Where general normalising flows require O(d3) work for the log-determinant-Jacobian, this approach reduces the cost to O(d2) or O(d).The general-flow cost includes backpropagation through the log-determinant-Jacobian.Source: §2.2.3.5 · p. 32

2.2.4 Latent ODEs

A latent ODE uses a VAE to model distributions over irregularly sampled time series, and the construction extends to partially observed data.The presentation assumes fully observed time series for ease of presentation.Source: §2.2.4 · p. 33, Figure 2.3 · p. 33

4 (Neural Stochastic Differential Equations)

This section surveys neural ODE models and their design, covering latent ODEs for time-series generation, their VAE-based construction, training, and sampling, as well as residual-network variants and parameterisations. It also examines how augmentation and time-dependent parameterisations affect expressivity, including universal-approximation results and the topology-preserving limitation of unaugmented ODE flows.

4 (Neural Stochastic Differential Equations)

A latent ODE VAE maps a sampled latent input to an initial state, evolves it by an ODE, and uses each evolving state to parameterise an output distribution.The encoder represents a time series with a diagonal-covariance Gaussian; the standard VAE objective trains reconstruction and matches the initial latent distribution to a known sampling distribution.Source: §4 · pp. 33–34

2.2.4.1 Examples

The trained latent ODE produces excellent decaying-oscillator samples over an interval four times longer than its training interval.Figure 2.4 shows sample quality increasing during training, including samples from the untrained and fully trained models.Source: §2.2.4.1 · pp. 35–36, Figure 2.4 · p. 35
Continuous-time latent ODEs handle irregular input timestamps across batch elements and generate outputs at all times, unlike RNNs restricted to prespecified timestamps.The example's observations are irregularly sampled and differ in timestamp placement between batch elements.Source: §2.2.4.1 · pp. 35–36

2.2.4.2 Sequence-to-sequence models

For forecasting, an RNN or neural CDE encoder can feed a neural ODE or SDE decoder; output distributions may be chosen as simple losses or expressive uncertainty models.Gaussian or Laplace distributions suit point statistics, while normalising flows can provide more expressive outputs; [Den+21] also replace the latent ODE with a latent SDE.Source: §4 · p. 35, §2.2.4.2 · p. 36

2.2.5.1 Rotational vector fields

Rotational residual-network updates have purely imaginary Jacobian eigenvalues and are largely immune to vanishing or exploding gradients, but volume preservation may reduce expressivity.The cited construction replaces the residual-network forward pass with a semi-implicit Euler discretisation; the limitation can be partially mitigated in higher dimensions.Source: §2.2.5 · p. 36, §2.2.5.1 · p. 37

2.2.5.2 Momentum residual networks

Momentum residual networks [San+21] are reversible, allowing intermediate activations to be recomputed during backpropagation for improved memory efficiency at extra computational cost.Their reversibility follows from the reversible semi-implicit Euler method; reversible solvers also support time- and memory-efficient backpropagation without requiring this second-order structure.Source: §2.2.5.2 · pp. 37–38

2.3 Choice of parameterisation

Alternative integration schemes include linear multistep and IMEX methods, while parameterisation practice commonly uses feedforward or convolutional vector fields and often avoids normalisation.ReLU activations can work in practice despite the differentiability requirement for ODE backpropagation; initialising vector fields near zero often improves training, while good architecture and optimiser choices may still require hyperparameter search.Source: §2.2.5.3 · p. 38, §2.3.1 · p. 39, §2.3.1.1 · pp. 39–40

2.3.2 Non-autonomy

Non-autonomous vector fields can vary with continuous depth through stacked networks, spectral parameterisations, or parameters themselves generated by a neural ODE.Time can be concatenated with the state as input; spectral bases may use Chebyshev polynomials or truncated Fourier bases.Source: §2.3.2.1 · p. 41, §2.3.2.2 · p. 41, §2.3.2.3 · p. 41

2.3.2.3 Hypernetworks

A hypernetwork can evolve the parameters of an original neural ODE over time, with both differential equations concatenated and solved as one system.The resulting system is still a neural ODE, with a particular structure in its vector field.Source: §2.3.2.3 · p. 42

2.3.2.4 Variant layers

A variant time-dependent layer injects time at each MLP layer rather than concatenating it with the hidden state, resembling GRU and LSTM gating.The example replaces affine layers in an MLP with time-dependent layers, inspired by [Gra+19].Source: §2.3.2.4 · p. 42

2.3.2.5 Enforcing autonomy

For time-series problems, including latent ODEs, one may assume the underlying dynamics are autonomous and remove time as an input.The passage notes that this assumption may also apply to neural CDEs and neural SDEs.Source: §2.3.2.5 · p. 42

2.3.3 Augmentation

Augmentation lifts inputs into a higher-dimensional hidden state and can dramatically improve model performance by avoiding topology-preservation constraints.The choice of zero, learned, or affine augmentation is usually unimportant; CNFs cannot use augmentation because their operations must be bijective. The lifted state relaxes the Markov property, while the Markov setting is often restrictive.Source: §2.3.3 · p. 43

2.3.3.1 Second-order-augmentation

Second-order augmentation adds velocity dimensions, a choice suited to modelling oscillatory dynamical systems [Nor+20].The cited variant takes the latent dimension to be twice the input dimension and structures the vector field so the extra dimensions represent velocities.Source: §2.3.3.1 · p. 44

2.3.3.2 Augmenting rotational vector fields

Augmentation can make a classification impossible for unaugmented ODEs feasible: the projected points {a, c} and {b} become linearly separable after arbitrarily little evolution.The unaugmented construction cannot linearly separate these points because the required flow would violate the non-crossing property.Source: §2.3.3.2 · p. 44

2.4.1 ‘Unaugmented’ neural ODEs are not universal approximators

Unaugmented neural ODEs are not universal approximators because their continuous flows preserve input topology and never cross.In the one-dimensional example, separating {a, c} from {b} would require trajectories to cross; the figure illustrates this obstruction. Higher-dimensional counterexamples use nested shells [DDT19].Source: §2.4.1 · p. 45, Figure 2.5 · p. 45

2.4.2 ‘Augmented’ Neural ODEs are universal

Augmented neural ODEs can approximate arbitrary continuous functions, including when their vector fields are not themselves universal approximators.With universal vector fields, the latent dimension is fixed and complexity comes from the vector field; without them, latent dimensionality may grow arbitrarily and complexity comes from affine maps. The result for non-universal vector fields is new here.Source: §2.4.2.1 · pp. 45–46, §2.4.2.2 · p. 46, §2.4.2.3 · p. 46

Neural Controlled Differential Equations

This section presents neural controlled differential equations as continuous-time models for sequential, irregular, and long time series, examining their expressivity and theoretical properties. It also covers computational and architectural choices—including vector fields and interpolation schemes—and situates this work alongside related rough-path and neural-SDE research.

3.1.1 Controlled differential equations

A CDE is an operator from input paths to output paths; with a Lipschitz vector field and bounded-variation control, its solution exists uniquely.Choosing the vector field appropriately lets a CDE compute specific functions of its control.Source: §3.1.1 · p. 50, Existence and uniqueness · p. 51

3.1.2 Neural vector fields

Neural CDEs [Kid+20a] use a learned vector field to update hidden state continuously from observations, with affine readouts at each time or at the terminal time.The hidden state reflects an evolving belief about the system, updated as observations arrive.Source: §3.1.2 · p. 52

3.1.3 Solving CDEs

When the control path is differentiable, a CDE can be reduced to an ODE and solved and trained using neural ODE techniques.In practice, most software libraries do not support the alternative CDE-specific discretisation, with Diffrax noted as an exception.Source: §3.1.3 · p. 53

3.1.4 Application to regular time series

For regularly sampled time series, an interpolated path with a time channel can drive a neural CDE; spiral-classification predictions update as input arrives and achieve perfect accuracy.Including time supports expressivity: the model can compute sequence length, which is impossible for a zero-valued input path without that channel.Source: §3.1.4 · p. 53, §3.1.4.1 · p. 54, §3.1.4.2 · pp. 54–55

3.1.5.1 Universal approximation

With the extra time-like variable included, an affine terminal-state readout makes neural CDEs universal approximators from sequences in R^dx to R.The stated universal approximation result depends on following the formulation carefully and including this variable.Source: §3.1.5.1 · p. 55

3.1.5.2 Continuous-time updates

Neural CDEs update continuously, with computational work scaling with the gap between observations, unlike RNNs’ equal-sized update steps.An adaptive-step solver can detect the dynamics’ complexity and take appropriately sized numerical steps.Source: §3.1.5.2 · p. 56

3.1.5.3 Memory efficient backpropagation

Neural CDE backpropagation uses O(H + T) memory, versus O(HT) for an RNN evaluated on a time series of length T.The reduction uses optimise-then-discretise backpropagation and stores the underlying data x in memory.Source: §3.1.5.3 · pp. 56–57

3.1.6.1 Rough path theory

CDE theory extends to rough driving paths beyond bounded variation, yielding rough differential equations and a framework that relates ODEs, CDEs, RDEs, and SDEs.Rough path theory provides a pathwise framework that can unify these equation types.Source: §3.1.6.1 · p. 57

3.2.1 Irregular time series

A continuous interpolation lets neural CDEs treat regular and irregular time series alike, but represents the input rather than imputing missing values.The interpolation choice, including its time parameterization, depends on the problem. Neural CDEs can also serve as discriminators for continuous-time paths generated by SDEs.Source: §3.2.1 · p. 58, §3.2.1.4 · p. 61, §3.2.4 · p. 63

3.2.1.1 Missingness as a channel

Observation frequency can carry information, so appending cumulative observation counts as input channels lets the neural CDE’s vector field respond to observation rate and channel-specific missingness.For partially observed data, separate count channels track observations in each original channel.Source: §3.2.1.1 · p. 59, §3.2.1.2 · p. 59

3.2.1.3 Batching irregular data and choice of sj

Setting the interpolation times s_j=j gives batched neural CDEs a shared integration start and simplifies path storage, though adaptive solvers can couple batch elements through step-size selection.The approach avoids needing solver support for different integration intervals across batch elements.Source: §3.2.1.3 · p. 60

3.2.2 RNNs are discretised neural CDEs

Euler discretization makes neural CDEs RNN-like, and RNN updates can in turn be represented as neural CDEs [Kid+20a].The continuous-time GRU includes hidden-state decay, which the text identifies as explaining why GRUs and LSTMs struggle with long-term dependencies.Source: §3.2.2 · p. 61, §3.2.2.1 · p. 61, §3.2.2.2 · p. 61

3.2.3 Long time series and rough differential equations

Neural CDEs can degrade on very long time series; the log-ODE method addresses this by taking large integration steps while incorporating sub-step information, yielding neural rough differential equations [Mor+21b].The text relates this method’s inclusion of sub-step information to the difference between Euler–Maruyama and Milstein methods.Source: §3.2.3 · p. 62

3.3.1 Universal approximation

An affine map of a neural CDE’s terminal value can universally approximate functions from sequences in R^dx to R.A sufficiently large CDE can compute a truncated basis of continuous functions of its input, which the affine map combines to approximate any continuous function.Source: §3.3.1 · p. 63

3.3.2 Comparison to alternative ODE models

Neural CDEs can exactly represent equations whose integrand is hθ(y(s), x(s)), but the converse representation does not hold.The text attributes this asymmetry to a neural CDE’s ability to represent the identity function between paths, which the alternative form cannot do.Source: §3.3.2 · p. 64

3.3.3 Invariances

Neural CDEs are invariant to translating the input and to changing its traversal speed; the initial value network restores translation sensitivity, while speed information must be encoded as a channel.The terminal value is unchanged under differentiable increasing reparameterizations that preserve the endpoints.Source: §3.3.3.1 · p. 64, §3.3.3.2 · p. 64

3.4.2 State-control-vector field interactions

The final state-control-vector-field layer can cost O(dh dydx) and bottleneck computation; sparse versions around 1% density often work well, while rank-one representations perform poorly.The text identifies reduced-parameter parameterisations as a topic for further investigation.Source: §3.4.2 · pp. 65–66

3.4.3 Multi-layer neural CDEs

Stacking CDEs can increase model capacity, analogously to multi-layer RNNs, but the second vector field’s output dimension dy1 × dy2 makes the approach computationally expensive.A later CDE can be driven by the hidden state produced by the preceding CDE.Source: §3.4.3 · p. 66

3.5.1.1 Measurability

Continuous measurability uses only past or present observations, but is relatively difficult to construct and is needed for online predictions between observations or before the final observation with missing data.Many use cases can accept weaker measurability.Source: §3.5.1.1 · pp. 67–68
Discrete measurability permits one-observation lookahead and suits online predictions only at observations when data are complete; non-measurable schemes can use the whole series and suit inference after it is available.Linear interpolation is discretely measurable, while natural cubic splines are non-measurable.Source: §3.5.1.1 · pp. 68–69

3.5.1.2 Smoothness

Smoother interpolation yields easier-to-integrate dynamics and better computational efficiency; globally smooth schemes are preferable, while piecewise smooth ones can be tolerated.Hermite cubic splines with backward differences are continuously differentiable, whereas linear interpolation is only piecewise continuously differentiable and can challenge adaptive solvers, though it can be cheaper with fixed-step solvers.Source: §3.5.1.2 · p. 69, §3.5.3.1 · p. 70, §3.5.3.2 · p. 70

3.5.2 Choice of interpolation points

Interpolation-point spacing roughly sets computational work between observations; choosing sj = tj is often reasonable, while fixed-step integration with sj = j and unit steps reduces a neural CDE to an RNN.In continuous time, the integral is invariant to the choices of T and sj, but this does not fully carry over to numerical discretisation.Source: §3.5.2 · p. 69

3.6 Comments

Natural cubic splines are very smooth but non-measurable and produce slightly worse neural CDE results than other schemes [Mor+21a]; neural CDEs were introduced in [Kid+20a], and rectilinear interpolation is due to [LLN13].The original neural CDE work used natural cubic splines, while [Mor+21a] investigated interpolation choices.Source: §3.5.3.4 · pp. 71–72, §3.6 · p. 72

3.5.3.5 Overall

Hermite cubic splines with backward differences are preferred when their measurability suffices; otherwise, rectilinear interpolation supports waiting for incoming data, including partially observed data.Rectilinear interpolation is piecewise differentiable, so solvers must handle its jumps appropriately.Source: §3.5.3.3 · p. 71, §3.5.3.5 · p. 72

3.6 Comments

Neural RDEs applying neural CDEs to long time series were introduced in.Source: §3.6 · p. 73
Rough-path theoretic ideas also frame RNNs as a kernel method in [Fer+21].Source: §3.6 · p. 73
Applications of neural SDE training were introduced in [Kid+21b] [Kid+21a].Source: §3.6 · p. 73
Discussion of architectural choices, gating, and sparsity is identified as new in this work.Source: §3.6 · p. 73

Neural Stochastic Differential Equations

This section presents neural SDEs for learning distributions over continuous-time paths or terminal values, covering their construction, parameterization, and training. It discusses Brownian-driven dynamics, latent and adversarial approaches, discriminator regularization and matching objectives, memory-efficient training, and examples involving stochastic processes, an oscillator, and the chaotic Lorenz system.

4.1.1 Stochastic differential equations

SDEs model continuous-time random phenomena, and classical calibration matches expectations of prespecified witness or payoff functions between generated paths and observed data.For analytically tractable models such as Black–Scholes, the calibration objective may be computed explicitly.Source: §4.1.1 · pp. 74–75

4.1.2 Generative and recurrent structure

Neural SDEs model distributions over paths, unlike neural CDEs, which model functions of input paths; discretised neural SDEs correspond to RNNs driven by random noise.The terminal value can also be treated as a static generative sample, with randomness injected by Brownian motion rather than only by the initial condition.Source: §4.1.2 · pp. 76–77

4.2 Construction

A neural SDE combines neural initialisation, drift, diffusion, and readout to model data-path distributions, with hidden state retained because the output need not be Markov.A separate initial noise variable is needed because Brownian noise does not determine the initial condition.Source: §4.2 · p. 78

4.3 Training criteria

Neural SDEs can be trained by matching the model and data path distributions using either Wasserstein distance for SDE-GANs or KL divergence for latent SDEs.These are the two main training criteria described for fitting the model law to the empirical data law.Source: §4.3 · p. 79

4.3.1 SDE-GANs

SDE-GANs train adversarially with a neural CDE discriminator that scores generated continuous paths, extending classical calibration from fixed payoff functions to a learned discriminator.The discriminator needs to be Lipschitz, and recurrent structure requires care when applying methods such as gradient penalty or spectral normalisation.Source: §4.3.1 · pp. 79–81, §4.3.1.1 · p. 81

4.3.1.1 Lipschitz regularisation

Wasserstein GANs require a Lipschitz discriminator, and recurrent SDE structure calls for care in applying weight clipping, gradient penalty, or spectral normalisation.The discussion points to recurrent-specific implementation considerations for these regularisation methods.Source: §4.3.1.1 · p. 81

4.3.1.2 Discretised observations

For sparse observations, the SDE-GAN discriminator interpolates generated and true data paths, preserving modelling and discriminatory power; dense observations allow simple linear interpolation.In the dense regime, the generator and discriminator may be concatenated into a single SDE solve.Source: §4.3.1.2 · p. 81, §4.3.1.3 · p. 82

4.3.1.3 Single SDE solve

Combining the generator and discriminator into one SDE solve improves training memory from O(H + T) to O(H), without recording the solution or output paths.This applies in the dense-data regime and is relevant to optimise-then-discretise training or reversible solvers.Source: §4.3.1.3 · p. 82

4.3.2 Latent SDEs

Latent SDEs use a data-conditioned auxiliary SDE to autoencode observations, then minimize KL divergence against the generative model; training solves only the auxiliary SDE.The auxiliary SDE is interpreted as a posterior SDE in a variational autoencoder with a learned prior.Source: §4.3.2 · p. 83

Single SDE solve

The affine decoder’s fixed-variance Gaussian likelihood is a simplifying choice: other distributions and parameterizations can be used with their log-likelihood objective.The Gaussian choice corresponds to an L2 loss for the first and third terms of the training objective.Source: Single SDE solve · p. 84

4.3.3 Comparisons and combinations

SDE-GANs are harder to train but have greater modeling capacity, while latent SDEs train more easily but can yield worse models with too-small diffusion; combining them can offset these weaknesses.The hybrid approach is illustrated for modeling a Lorenz system elsewhere in the paper.Source: §4.3.3 · p. 84

4.4 Choice of parameterisation

SDE-GAN training is unstable; Adadelta or RMSprop and stochastic weight averaging can help, whereas latent SDEs train readily with Adam, though good hyperparameter choices remain an open problem.Adadelta can outperform SGD, Adam, and Adam with β1 = 0; averaging generator and discriminator weights can reduce oscillatory min-max training behavior.Source: §4.4 · p. 84, §4.4.1.1 · p. 85, §4.4.1.2 · p. 85

4.4.2.1 Generator

Neural SDE generators can approximate any Markov SDE under suitable function classes, but non-Markov approximation via hidden state lacks a formal guarantee; latent-SDE KL computation motivates diagonal diffusion.Diagonal diffusion makes inversion cheaper, and its entries can be bounded away from zero for numerical stability; Brownian noise can also be replaced or supplemented by fractional Brownian motion or Lévy processes, including counting-process jumps.Source: §4.4.2.1 · p. 86, Approximation properties · p. 87

4.4.3 Lipschitz regularisation

A recurrent SDE-GAN discriminator can have Lipschitz constant O(λ^T), so careful clipping and smooth LipSwish activations constrain its vector fields to Lipschitz constant at most one.Clipping each linear map’s entries to [−1/b, 1/b] enforces the needed bound, while LipSwish is smooth and has Lipschitz constant one.Source: §4.4.3.1 · pp. 87–88, Careful clipping · p. 88, Overall · p. 88

4.4.3.3 Gradient penalty

Gradient penalty works but is usually less attractive than careful clipping because it requires a double backward, can degrade optimise-then-discretise training, and may increase training time by about 50%.The discretization error can be sufficient to degrade or destroy training, and the path-valued derivative is defined with respect to the numerically discretized solution.Source: §4.4.3.3 · p. 89, Disadvantages · p. 89

4.5 Examples

SDE-GAN examples include Brownian motion and a time-dependent Ornstein–Uhlenbeck process with drift and diffusion, which are difficult for pure-drift latent ODEs or almost-pure-diffusion CTFPs [Den+20].The examples also include samples from a two-dimensional damped harmonic oscillator over t ∈ [0, 100].Source: Brownian motion · p. 89, Figure 4.3 · p. 90, Time-dependent Ornstein–Uhlenbeck… · p. 90

Damped harmonic oscillator

The combined latent SDE/SDE-GAN, trained by alternating their steps, correctly learns the distribution of a chaotic multidimensional Lorenz time series.Figure 4.6 shows evolving samples from the Lorenz attractor and the trained model.Source: Figure 4.6 · p. 91, Damped harmonic oscillator · p. 91
The damped-oscillator latent SDE matches drift and extrapolates beyond observations, but retains excessive diffusion despite zero true diffusion, illustrating neural SDEs’ practical and theoretical inclusion of neural ODEs.Source: Damped harmonic oscillator · p. 91
For Brownian-motion and Ornstein–Uhlenbeck data observed at integer times with 70% probability, continuous-time models need no special handling and can be queried anywhere in time.Source: Damped harmonic oscillator · p. 92

4.6 Comments

Beyond Wasserstein or KL matching, calibration with fixed statistics gives an MMD formulation, but feature-map choice matters; the signature transform is cited as a good choice [KL21].Source: §4.6 · p. 92
[Bri+20; Gie+20] match only finite-dimensional marginals, [CKT20] adds prespecified payoff structure to the discriminator, and [CRW21] uses maximum likelihood for Markov neural SDEs.Source: §4.6 · p. 92

Numerical Solutions of Neural Differential Equations

This section surveys numerical methods for neural differential equations, covering how ODE, CDE, and SDE solvers are differentiated and how solver choices, adjoints, reversibility, and regularization affect computation and gradients. It also discusses adaptive and jump-aware integration, Brownian path simulation and sampling, learned hypersolvers, and NDE software, including comparisons and proofs concerning numerical methods.

5.1 Backpropagation through ODES

Discretise-then-optimise is generally preferred for its accurate gradients and speed, unless memory constraints favor reversible solvers; optimise-then-discretise is typically least favored.It differentiates the discretised model actually solved, but stores every solver operation, using O(HT) memory; checkpointing can reduce that burden by recording selected forward values.Source: §5.1 · p. 94, §5.1.1.1 · p. 95, Speed · p. 95

5.1.1.3 Checkpointing

Checkpointing limits memory use by recording selected forward-pass values: it can reconstruct solver values for discretise-then-optimise or reset accumulated backward-pass truncation error for continuous adjoints.For continuous adjoints, checkpointing raises memory use from O(H) to O(H + C), where C is the number of checkpoints.Source: §5.1.1.3 · p. 96, §5.1.2.5 · p. 101

5.1.2 Optimise-then-discretise

The continuous adjoint method differentiates the idealised continuous-time model via backward-in-time adjoint equations, reducing memory from O(HT) to O(H), independent of T.It can treat the solver as a black box, but recomputation adds computational cost and numerical errors can make gradients less accurate, slow training, hurt performance, or cause failure.Source: §5.1.2 · pp. 96–97, §5.1.2.2 · p. 98, §5.1.2.3 · pp. 98–99

5.1.2.4 Interpolated adjoints

Interpolated adjoints record and interpolate selected forward solution values, avoiding backward recomputation; they use modest memory and can address numerical errors and stability issues.[Kim+21a] report that optimise-then-discretise adjoints failed on a stiff equation while interpolated adjoints and discretise-then-optimise succeeded.Source: §5.1.2.4 · pp. 99–100

5.1.3 Reversible ODE solvers

Reversible ODE solvers combine memory efficiency with accurate gradients, at the cost of extra computation to recompute the forward solution during backpropagation.At the time described, known general methods include reversible Heun and asynchronous leapfrog; their low order and poor stability can impair solutions for structured differential equations.Source: §5.1.3 · p. 101

5.1.4 Forward sensitivity

Forward sensitivities can use either discretise-then-optimise or optimise-then-discretise, but are typically less efficient than reverse-mode autodifferentiation for machine-learning problems with many parameters.[Rac+20a] report finding forward sensitivity useful only on problems with fewer than 100 parameters.Source: §5.1.4 · pp. 101–102

5.2.2 Optimise-then-discretise for CDEs

Discretise-then-optimise differentiates through solver operations, while CDE optimise-then-discretise can reduce to an ODE adjoint or solve a backward CDE and recompute states.Differentiating solver operations is typically done with solvers in an autodifferentiable framework.Source: §5.2.2 · pp. 102–103, §5.2.1 · p. 102

Rough path theory

For SDEs, the backward adjoint uses the forward pass’s Brownian motion; rough path theory gives pathwise meaning, and Stratonovich avoids costly Itô corrections.The backward equation’s vector-Jacobian products can be computed using autodifferentiation.Source: §5.2.3 · p. 103, Rough path theory · p. 104

5.2.4 Reversible differential equation solvers

CDEs can use reversible ODE solvers after reduction to ODEs, while the reversible Heun method is the one known reversible solver for SDEs.Source: §5.2.4 · p. 104

5.3.1.1 General principles

Implicit solvers are costly and often unnecessary for neural differential equations, though structured vector fields or multiple timescales can make stiffness—and implicit methods—relevant.Neural vector fields are unstructured, so specialized equation-specific solvers often do not apply; training to reduce loss can also avoid stiffness in typical settings.Source: §5.3.1 · p. 104, §5.3.1.1 · p. 105
Fixed and adaptive step solvers are both reasonable; adaptivity can take larger steps on slowly varying dense inputs, while training on one discretisation may impair performance with others.Adaptive solver cost varies and typically rises as model complexity increases during training.Source: §5.3.1.1 · pp. 105–106

5.3.1.3 SDEs

For continuous-model fitting, higher-order Dormand–Prince is often preferred for ODEs and CDEs, while SDEs commonly use Euler–Maruyama for Itô or Heun for Stratonovich.A single Euler–Maruyama step makes the conditional distribution y(t_j+1)|y(t_j) Gaussian, which is relatively inexpressive.Source: §5.3.1.1 · pp. 106–107, §5.3.1.3 · p. 107

5.3.2 Reversible solvers

Reversible solvers cost approximately six forward operations per step, versus four for discretise-then-optimise and five for optimise-then-discretise, while recovering discretise-then-optimise gradients.With suitable solver structure, the local forward operation can be omitted, saving one forward operation.Source: §5.3.2 · p. 108

5.3.2.1 Analytic and algebraic reversibility

Analytic reversibility may require an approximate, expensive fixed-point solve, whereas algebraic reversibility gives a closed-form inverse and may be symmetric.A symmetric solver uses the same step form in reverse with a negative time step.Source: §5.3.2.1 · p. 109

5.3.2.2 Reversible Heun method

The reversible Heun method is a general non-symplectic algebraically reversible SDE solver, second-order for ODEs and strongly convergent of order 1/2 for SDEs, rising to order 1 with additive noise [Kid+21a].Its ODE stability region is [−i, i], but it uses one evaluation per step and may be more computationally efficient than traditional Heun.Source: §5.3.2.2 · pp. 109–111

5.3.2.3 Asynchronous leapfrog method

The asynchronous leapfrog method is a second-order, algebraically reversible ODE/CDE solver with stability region [−i, i], but it does not extend reliably to SDEs [Mut13].Its reverse pass can omit an extra forward operation by differentiating the reverse pass; the analogous SDE procedure lacks clear convergence theory and performs relatively poorly empirically.Source: §5.3.2.3 · pp. 111–113

5.3.2.4 Symplectic solvers

The leapfrog/midpoint integrator is symmetric and algebraically reversible for general first-order systems, but its multistep form limits adaptive stepping and has poor stability.Semi-implicit Euler is also highlighted for its popularity in deep-learning papers.Source: §5.3.2.4 · p. 114

5.3.3 Solving vector fields with jumps

Informing an adaptive ODE solver of vector-field jump locations avoids repeated error-control slowdowns and can prevent an order-of-magnitude slowdown.Separate solver calls can increase memory cost by a factor of n, though this may be desirable as checkpointing.Source: §5.3.3 · p. 115

5.3.4 Hypersolvers

80 Dormand–Prince steps can be replaced by 2 HyperHeun steps without degrading accuracy in a continuous normalising flow.Hypersolvers learn an error correction and are trained using access to the true solution, which can be approximated with a high-order solver, small steps, or tight tolerances; CDE and SDE extensions remain unexplored.Source: §5.3.4 · p. 116, Applications · p. 117

Applications

Hypersolvers are generally useful for speeding inference rather than training, because the neural ODE changes during training and makes the learned correction a moving target.Source: Applications · p. 117

5.4.1.1 Weight decay

Weight decay may improve model performance and, by shrinking neural vector fields toward zero, make them numerically easier to integrate.Source: §5.4.1.1 · p. 117

5.4.1.2 Temporal regularisation

For non-time-series applications such as continuous normalising flows, temporal regularisation can sample the integration endpoints from a distribution instead of fixing them.Source: §5.4.1.2 · p. 117
Temporal regularisation perturbs the terminal time while keeping τ = 0 fixed, whereas additive noise after each step encourages robustness to small perturbations [OVV20].The added noise must be fixed rather than learned, or training will shrink it to zero.Source: §5.4.1.2 · p. 118, §5.4.1.3 · p. 118

5.4.1.4 Regularising higher-order derivatives

For a q-th order numerical ODE solver, the per-step error is determined by the (q+1)-th total derivative; regularising it can minimise numerical errors and promote easy-to-integrate dynamics [Kel+21].A Jacobian-vector product costs 2.5 times the corresponding forward evaluation, so nesting K evaluations costs 2.5^K = O(exp(K)); exploiting higher-order derivative structure can reduce this to O(K^2) [Kel+21, Appendix A; GW08, Chapter 13].Source: §5.4.1.4 · p. 118

Continuous normalising flows

For continuous normalising flows, Jacobian terms in this regularisation are cheap because the Jacobian is already computed, avoiding additional autodifferentiation calls.Source: Continuous normalising flows · p. 119

5.4.2 Exploiting the structure of adaptive step size controllers

Adaptive Runge–Kutta controllers accept a step when the error ratio is at most 1 and reject it otherwise, with the chosen norm setting channels’ relative importance.Source: §5.4.2 · p. 120

5.4.2.1 Not-an-ODE and adjoint seminorms

40%–62% lower backward-pass cost for neural CDEs and Hamiltonian neural networks, and 5% for CNFs, were reported using adjoint seminorms.They use generally larger backward-pass steps without harming model performance.Source: §5.4.2.1 · pp. 121–122

5.4.2.2 Non-backpropagation through adaptive step size controllers

Backpropagating through adaptive step-size selection adds computation and can cause gradient pathologies, so implementations commonly treat the selected step size as constant.The torchdiffeq and Diffrax libraries do not backpropagate through step-size selection.Source: §5.4.2.2 · p. 122

5.4.2.3 Regularising error estimates

Regularising solver error estimates is computationally almost free because the per-step error is already computed, but requires discretise-then-optimise or a reversible solver.Source: §5.4.2.3 · p. 123

5.5 Numerical simulation of Brownian motion

Adaptive SDE backpropagation may query the same Brownian path at new times; storing every sample enables O(1) queries but costs O(dwT) memory, motivating the Virtual Brownian Tree, while Brownian Interval is usually preferred in practice.Source: §5.5 · p. 123, §5.5.1 · p. 124, §5.5.2 · p. 124

5.5.2 Virtual Brownian Tree

The Virtual Brownian Tree reconstructs the same Brownian motion from a root seed without storing individual samples, using O(1) memory and approximate O(log(1/ε))-time queries.Queries discretise endpoints to nearby dyadic points, so the resulting Brownian increments are approximate.Source: §5.5.2 · p. 125

5.5.3 Brownian Interval

The Brownian Interval answers interval queries with exact samples and O(1) query times.Its query-dependent interval tree uses an LRU cache; average time is O(1), worst-case time is O(log(1/h)), GPU memory is O(1), and tree storage uses O(T) CPU memory.Source: §5.5.3 · p. 125, §5.5.3.1 · p. 127

5.5.3.1 Overview

The section’s overview uses a query-dependent tree of interval-seed pairs, with an LRU cache to make exact Brownian queries fast when successive SDE solver queries are close.The tree structure grows with queries and uses O(T) CPU memory for a solve on [0, T].Source: §5.5.3.1 · pp. 126–127

5.5.3.2 Algorithmic definitions and further discussion

Interval queries require one query for w(s, t), rather than two point queries, and support interval-based higher-order quantities such as Lévy areas.An interval-based Virtual Brownian Tree may be inefficient when ε is small and solver steps are much larger than ε.Source: §5.5.3.2 · p. 128

5.6 Software

NDE software packages provide standardized numerical solving and training tools, and Diffrax unifies ODE, CDE, and SDE solving while supporting batches over different integration regions.Options include Diffrax in JAX, torchdiffeq, torchcde, and torchsde in PyTorch, and DifferentialEquations.jl in Julia.Source: §5.6 · pp. 128–130

5.7 Comments

The broader comparison of discretise-then-optimise, optimise-then-discretise, and reversible solvers is new, as are the presented proofs of optimise-then-discretise for ODEs, CDEs, and SDEs.The section also identifies a new, more efficient backward step for asynchronous leapfrog and describes its comparison of software libraries as new.Source: §5.7 · pp. 130–131

Miscellanea

This section surveys symbolic regression of learned dynamics alongside limitations of neural differential equations and broader implicit-model approaches. It covers deep equilibrium and implicit layers, as well as multiple shooting for continually updated ODE forecasts.

6.1.1 Introduction to symbolic regression

Symbolic regression seeks shallow expressions from data, but expression spaces are difficult to traverse and grow intractable beyond simple problems, while trained deep-learning models can remain opaque.Constants may be differentiably optimised, but transitions between expressions are usually searched via genetic algorithms; the passage estimates (2n)! (n+1)!n! binary trees with n vertices.Source: §6.1.1 · p. 132

6.1.2 Symbolic regression for dynamical systems

Neural ODE fitting can learn dynamics from sparse state observations without paired derivatives, after which symbolic regression of the learned vector field removes SINDy’s preselected-feature restriction.SINDy [BPK16] assumes paired observations of y and dy/dt and a linear combination of preselected features; the neural ODE approach instead fits trajectories by backpropagating through a numerical solution.Source: §6.1.2 · p. 133

6.1.3 Example

In a nonlinear-oscillator example, a trained neural ODE learned the problem structure, and symbolic regression followed by gradient refinement and rounding produced the desired vector field.The neural ODE reconstruction aligned closely with the data; the example uses regularised evolution for symbolic regression and then optimises constants against the original data.Source: §6.1.3 · pp. 135–136, Figure 6.1 · p. 135

6.2 Limitations of neural differential equations

NDEs require more data than classical differential equations, and higher-order solvers can make them slow to evaluate or train.A few hundred samples is described as an optimistic lower bound; cheaper low-order solvers can reduce cost, and solver choice can vary between bulk training and fine-tuning or inference.Source: §6.2.1 · p. 136, §6.2.2 · pp. 136–137

6.2.3 Other discretised architectures

NDEs are one successful paradigm for constructing discretised architectures, but U-Nets and Transformers lack obvious NDE descriptions, although Transformers have a continuous theory [Ram+20b].Source: §6.2.3 · p. 137

6.3 Beyond neural differential equations: deep implicit layers

Deep implicit models specify outputs by conditions, commonly approximate their solutions numerically, and permit solver-based or implicit-function-theorem backpropagation; recent work suggests they may outperform explicit models [Flo+21] [Fun+21].Neural ODEs are themselves implicit models because their computational steps are left implicit.Source: §6.3 · pp. 137–138, §6.3.1 · p. 138

6.3.2 Deep equilibrium models

Deep equilibrium models [BKK19] define outputs through an implicit equation, often using a large network such as a Transformer solved by fixed-point iterations.Monotone Operator Equilibrium Models impose structure to ensure a unique solution, while regularisation may improve stability.Source: §6.3.2 · p. 138

6.3.3 Multiple shooting: DEQs meet NODEs

Multiple shooting reframes an ODE as an implicit problem over time intervals, whose ODE solves can run in parallel and reduce computation when few fixed-point steps suffice and a good initial guess is available.For training, prior interval solutions can initialise subsequent parameter-optimisation steps; fully observed Markov-system samples can also supply interval values.Source: §6.3.3 · p. 139
For continually updated ODE forecasts, each new forecast starts at the current time, and the old solution may initialise each b_j.Each forecast solves the ODE from the current time to a future time.Source: §6.3.3 · p. 139

6.3.4 Differentiable optimisation

Differentiable optimisation defines a layer by solving an optimisation problem; paired inputs and projections can train half-space constraints to approximate an unknown convex polytope.The half-space constraints are trainable parameters, and the polytope is the intersection of those half-spaces.Source: §6.3.4 · p. 140

6.4 Comments

The symbolic-regression discussion is new joint work with Miles Cranmer; neural differential equation limitations are folklore, and deep implicit layers are recent, popularised by.Source: §6.4 · p. 140

Conclusion … B.1. BACKGROUND

This section outlines open research directions for neural differential equations and reviews tools and concepts relevant to their study. It introduces neural rough differential equations for long time series, signatures and logsignatures for representing controlled-path information including event order, and supplementary background on the manifold hypothesis and MMD criteria.

7.1 Future directions

Most neural differential equation applications remain bespoke, with largely unstructured vector fields retrained for each use case; the authors anticipate more reusable models and long-term analyses.They compare this prospective development with the established literature on traditional differential equations and the study of deep-learning models such as GPT-3 and CLIP [Rad+21].Source: §7.1 · p. 141
Neural ODEs could extend to scientific and other tasks so far addressed only with non-neural ODEs, while newer neural CDEs and SDEs need practical advances and applications.For CDEs and SDEs, open directions include more expressive vector fields, efficient training, practical or hybrid applications; CDE connections to control and reinforcement learning, and SDE connections to score matching, continuous normalising flows, optimal transport, and Schrödinger bridges remain underexplored.Source: §7.1 · pp. 141–142
Symbolic regression for dynamical systems remains largely unguided: beyond established SINDy methods and variants, regularised evolution and deep symbolic regression are wide-open directions.The authors characterize current symbolic regression practice as more trial-and-error than principle-driven.Source: §7.1 · p. 142
Neural PDE research remains a major open direction: existing ideas include convolutional networks as parabolic-PDE discretisations, Fourier Neural Operators [Li+20b] [Li+20c], and initial neural stochastic PDE work [SLG21].Many ideas have yet to converge, in part because there are many PDE types to consider.Source: §7.1 · p. 142

A.1. AUTODIFFERENTIATION

Reverse-mode autodifferentiation is substantially cheaper than forward mode for vector inputs and scalar outputs, but requires intermediate values to be retained in memory.The comparison explains why neural networks typically use backpropagation, and notes that storing intermediate values can make memory a concern.Source: §A.1 · p. 145, Comparison · p. 145

A.2 Normalising flows

Normalising flows train a neural transformation from a normal random variable to data by maximum likelihood, yielding a generative model, but Jacobian log-determinants generally cost O(d^3).Architectures seek to exploit structure to reduce Jacobian computation while remaining expressive.Source: §A.2 · p. 146

A.4 Irregular time series

For irregular time series, observation times may differ and be informative, sequence lengths may vary, and missingness itself can carry information, so blindly imputing missing values can lose signal.A proposed approach maps observations, including missingness, injectively into a vector space usable by the model.Source: §A.4 · pp. 147–148

A.5 Miscellanea

MMD is a popular generative-model fitting criterion and underlies KID; its biased Monte Carlo estimate costs O(N), while the described unbiased estimate costs O(N^2).Replacing the pseudodistance’s finite-dimensional space with an infinite-dimensional Hilbert space makes it a true distance.Source: §A.5 · pp. 148–149

A.5. MISCELLANEA

The manifold hypothesis informally describes datasets as occupying a small, low-dimensional manifold within a much higher-dimensional space, illustrated by cat images among possible pixel arrays.Most points in the image space are described as non-cat pictures or random noise.Source: §A.5 · p. 149

Neural Rough Differential Equations

For long time series, where neural CDEs and RNNs tend to break down, neural rough differential equations use log-ODE integration steps much larger than the data sampling rate while adding sub-step information.The log-ODE-treated CDE is termed a rough differential equation, or the log-ODE method applied to neural CDEs.Source: Neural Rough Differential Equations · p. 150

B.1.1.1 Signatures

For a continuous bounded-variation path, its depth-M signature bundles iterated Riemann–Stieltjes integrals, with 1 included by convention.The text notes that weaker path conditions may also be admitted but uses bounded variation for its purposes.Source: §B.1.1.1 · p. 150, §B.1 · p. 151

B.1.1.2 Signatures as Taylor expansions

A signature appears in the CDE solution’s Taylor expansion, where higher-order expansions of the vector field introduce higher-order signature terms; it can therefore approximate the solution using information important over small time scales.The supplied expansion includes an O((t −a)3) remainder.Source: §B.1.1.2 · pp. 151–152

B.1.1.3 Logsignatures

The logsignature removes redundant signature terms; an efficient choice retains terms indexed by Lyndon words, while its first level gives path increments and its second gives signed Lévy area.Higher logsignature terms correspond to repeated areas.Source: §B.1.1.3 · p. 152, Figure B.1 · p. 153

B.1. BACKGROUND

(Log)signatures encode event order, connecting CDEs’ reliance on signatures and RNNs’ order-sensitive premise with the contrast that Transformers assume order is mostly unimportant.The text associates RNNs more often with time series and Transformers more often with natural language processing.Source: §B.1 · p. 153

B.1.2 The log-ODE method · B.2 Neural vector fields

The log-ODE method approximates a controlled differential equation by an ODE whose solution converges under the stated assumptions. The neural vector-field setup defines the time-series interpolation, parameterized fields, and initial state used to formulate and solve neural CDEs.

B.1.2 The log-ODE method

Under a Lipschitz vector field and a bounded-variation control, the log-ODE method gives ODE solutions whose terminal values converge to the CDE solution as M increases.The approximating vector field has output dimension R^dy×β(dx,M).Source: §B.1.2 · pp. 153–154
The log-ODE equation’s right-hand side is a matrix-vector product involving the approximating vector field evaluated at the ODE state.The matrix has dimension R^dy×β(dx,M).Source: §B.1.2 · p. 154
The exact approximating vector field is known but expensive to compute, motivating a method that avoids calculating it.Source: §B.1.2 · p. 154
At depth 1, the log-ODE vector field equals f; for piecewise linear x, the depth-1 approximation is exact and identical to equation (3.3).Source: §B.1.2 · p. 154

B.2 Neural vector fields

The neural CDE setup uses ordered time-stamped observations that may contain missing values, and interpolates them with continuous piecewise linear paths.Linear and rectilinear interpolation are examples that satisfy this requirement.Source: §B.2 · p. 154
Its parameterized vector field is a Lipschitz neural network, with hidden-state size dy as a hyperparameter, and a neural network ζθ maps the input to the initial state.Source: §B.2 · p. 154
The neural CDE is typically solved by reducing it to an ODE, using the initial state y(0) = ζθ(x(0)).Source: §B.2 · p. 154

B.2.1 Applying the log-ODE method … B.2.3 Efficacy on long time series

The log-ODE method represents a CDE using fewer logsignature intervals, allowing larger solver steps. Its speed, information content, and memory use depend on the interval and depth choices, and it is particularly helpful for long time series.

B.2.1 Applying the log-ODE method

The method reduces a CDE to a neural ODE driven by piecewise-linear logsignatures, using m ≪ n windows and tunable logsignature depth M.The window count and spacing are also hyperparameters; continuous piecewise-linear interpolation is used because efficient signature and logsignature algorithms are known for these paths [KL21].Source: §B.2.1 · p. 155

B.2.2 Discussion

Typical speed-ups are about tenfold because the shorter, more slowly varying logsignature path permits larger solver steps.Figure B.2 contrasts the quickly varying original path with the more slowly varying logsignature path, which needs fewer integration steps.Source: §B.2.2 · p. 156, Figure B.2 · p. 156
Implementation preprocesses each window into logsignatures, interpolates them as a piecewise-linear path, and solves a neural CDE; the steps are available in software libraries.The logsignature sequence acts as a selected binning that reduces data while retaining information important for solving the CDE.Source: §B.2.2 · p. 156
Directly parameterising fθ : R^dy → R^(dy×β(dx,N)) avoids the computational expense of computing the alternative lifted vector field.Source: §B.2.2 · p. 156

B.2. NEURAL VECTOR FIELDS

Larger intervals make training faster but less informative, whereas greater logsignature depth makes it slower but more informative; both are modelling hyperparameters.Accurate solution may require infeasibly large depth or small intervals, so the method uses a baked-in discretisation rather than treating these choices as implementation details.Source: §B.2 · p. 157

B.2.3 Efficacy on long time series

For long series, reducing sequence length helps address vanishing or exploding gradients and computation time, while preprocessing logsignatures need only occur once before training.Neural CDEs and RNNs are inherently serial across time and difficult to parallelise, unlike processing many channels.Source: §B.2.3 · p. 157
The log-ODE method reduces backpropagation memory from O(H + n) for continuous adjoint CDEs to O(H + m), with m ≪ n.Source: §B.2.3 · p. 157

B.2.4 Limitations … Proofs and Algorithms

The examples show that depth-3 neural RDEs outperform neural CDEs on long time-series tasks, while the method’s practicality depends on tuning truncation depth and step size and on the number of input channels. The section also relates log-ODE methods to standard numerical solvers and points to further work on signatures and neural RDEs.

B.2.4 Limitations

Truncation depth and step size are additional hyperparameters that substantially affect training time and memory usage and must be tuned.In the examples, logsignature depth is varied over {2, 3} and step size over {8, 128, 512}; depth 1 is identical to a standard neural CDE.Source: §B.2.4 · p. 158, §B.3 · p. 158
The log-ODE method is most feasible with few input channels because the number of log-signature channels β(d, M) grows exponentially in d, potentially saturating available parallelism.Source: §B.2.4 · p. 158

B.3 Examples

The examples use three BIDMC datasets to predict respiratory rate, heart rate, or oxygen saturation from regularly sampled PPG and ECG series.Each series has length 4,000, is sampled at 125 Hz, and has three channels including time; performance is evaluated using L2 loss.Source: §B.3 · p. 158
The long series required optimise-then-discretise backpropagation to avoid memory exhaustion; the experiments compare neural RDEs with neural CDE and ODE-RNN baselines.For neural CDEs, increased step sizes mean naive subsampling; for ODE-RNNs, time is folded into the feature dimension so each step sees adjacent time points.Source: §B.3 · p. 158
Depth-3 neural RDEs are the top performer on every task at every step size, reducing test loss by 30–59% versus the corresponding neural CDE.Table B.1 reports test-set L2 losses for the three BIDMC vital-sign prediction tasks, along with memory usage and training time.Source: §B.3 · p. 158, Table B.1 · p. 159

B.4. COMMENTS

Neural RDEs require roughly an order of magnitude less training time, while the ODE-RNN baseline performs poorly and uses significantly more memory.The improved test loss is attributed to better learning of long-term dependencies from reduced sequence length; rough-model performance improves as step size increases.Source: §B.4 · p. 159

B.4 Comments

Standard numerical SDE solvers such as Euler–Maruyama and Heun arise from ODE solvers applied to depth-1 log-ODE methods; Milstein is almost, but not exactly, equivalent at depth 2 [Lyo04, Section 7].Source: §B.4 · p. 159
Neural rough differential equations were introduced and applied to long time series in; signature resources include introductions [Rei17] [Rei18] and an efficient implementation [KL21].[Mor+21b] also provides further mathematical details, including convergence of the log-ODE method.Source: §B.4 · p. 159

C.1 Augmented neural ODEs are universal approximators even when their vector fields are not universal approximators … C.2.1 Neural CDEs are universal approximators

An augmented neural ODE can approximate any continuous map using a fixed vector field, with affine maps for augmentation and output. The section also comments on this construction and outlines neural CDE universal approximation for continuous paths and time series under interpolation conditions.

C.1 Augmented neural ODEs are universal approximators even when their vector fields are not universal approximators

For each augmentation dimension, a fixed vector field yields an augmented ODE map that is a universal approximator for continuous maps from R^d to R^d_o.The input and output maps are affine; the vector field need only have a unique solution.Source: §C.1 · p. 160

C.1. AUGMENTED NEURAL ODES ARE UNIVERSAL APPROXIMATORS EVEN WHEN THEIR VECTOR FIELDS ARE NOT UNIVERSAL APPROXIMATORS

Evaluating the constructed system produces scaled monomials, which can be combined into polynomial approximations of continuous functions.The proof invokes Stone–Weierstrass and uses affine maps to augment the input and combine monomials into a polynomial.Source: §C.1 · p. 161

C.1.1 Comments

In this construction, the only learned parameters are in the final affine map, and each component’s vector field depends on earlier components rather than itself.The text notes that these components are “only” integrals, raising a question about whether the construction is truly a neural ODE.Source: §C.1.1 · p. 161
The argument resembles universal approximation by wide networks or Fourier series, which also combine enough terms linearly.The passage cites for the wide-network comparison.Source: §C.1.1 · p. 162
The augmented ODE result is a special case of CDE universal approximation, using the input path z(t) = tx.The proof is described as a simplification and particular application of the CDE result.Source: §C.1.1 · p. 162

C.2.1 Neural CDEs are universal approximators

The CDE discussion first addresses universal approximation for continuous paths, then extends it to time series using interpolation conditions independent of the interpolation choice.The time-series extension requires the interpolation to satisfy certain conditions.Source: §C.2.1 · p. 162

C.2.1.1 Universal approximation with respect to paths … C.3.3.1 Fundamentals

These parts develop theoretical and methodological foundations for path-based neural differential equations, covering universal approximation by signatures and neural CDEs, comparisons with alternative ODE models, reparameterisation invariance, and backpropagation for ODEs, CDEs, and SDEs. They also introduce rough-path foundations and path lifts, including their role in specifying integration conventions and convergence.

C.2.1.1 Universal approximation with respect to paths

On compact path sets with uniqueness of signatures, the signature transform is dense in continuous functions of paths.Uniqueness means paths with the same start are distinguished by some finite-depth signature; the density result is the signature’s universal nonlinearity property.Source: §C.2.1.1 · pp. 163–164
Neural CDEs are dense in continuous maps on compact path sets with uniqueness of signatures.The theorem states density in C(K; R^do) for the CDE outputs defined in the section.Source: §C.2.1.1 · p. 164

C.2.1.2 Universal approximation with respect to time series

Bounded interpolation yields relatively compact interpolated paths when the time-series variation is uniformly bounded.This compactness result supports applying the path-level universal approximation theorem to time-series interpolations.Source: §C.2.1.2 · p. 167
Neural CDEs are dense in continuous functions of time series under a bounded, signature-unique interpolation and dense vector-field and initial-value classes.The theorem applies to irregular time series, including cases with missing observations; interpolation conditions depend on the time-series domain.Source: §C.2.1.2 · pp. 165, 168

C.2.2 Neural CDEs compared to alternative ODE models

Neural CDEs strictly include the alternative ODE model class described here, with time augmentation and a larger hidden state enabling the comparison result.The larger state can record the control in hidden channels, allowing implicit nonlinear dependence on it; time augmentation is needed to avoid restrictions from reparameterisation invariance.Source: §C.2.2 · pp. 169, 170, 171, 173

C.2.3 Reparameterisation invariance of CDEs

Under a differentiable increasing time change from [0, S] to [0, T], a CDE solution reparameterizes with its driving path and retains the same terminal value.The change of variables also applies to bounded-variation paths using equivalent change-of-variable formulae.Source: §C.2.3 · p. 173

C.2.4 Comments

The authors report that their discussion of interpolation schemes is new, noting that formalities of embedding discrete observations into continuous time are rarely addressed.They also note that they could not find a direct literature statement of Lemma C.8, though they describe it as easy to prove.Source: §C.2.4 · pp. 173–174

C.3 Backpropagation via optimise-then-discretise

The appendix proves optimise-then-discretise backpropagation separately for ODEs, CDEs, and SDEs, although these cases may be viewed as special cases of a general SDE result.The separate proofs are presented for pedagogical reasons.Source: §C.3 · p. 174

C.3.1 Optimise-then-discretise for ODEs

For ODEs, the optimise-then-discretise result gives adjoint differential equations for gradients with respect to the initial state and parameters.The proof is described as simpler and more precise than proofs the authors had typically seen.Source: §C.3.1 · p. 174

C.3.2 Optimise-then-discretise for CDEs

For CDEs, the adjoint process satisfies a backwards-in-time linear CDE, and the proof uses integration by parts in place of the product rule.The theorem assumes a Lipschitz, continuously differentiable vector field and a continuous bounded-variation control.Source: §C.3.2 · pp. 175–176

C.3.3 Optimise-then-discretise for SDEs

Classical SDE theory struggles to make sense of backward-in-time SDEs, motivating the use of rough-path theory for optimise-then-discretise backpropagation.The section begins by presenting the rough-path approach to SDEs.Source: §C.3.3 · p. 177

C.3.3.1 Fundamentals

For p in (2, 3), piecewise linear Brownian approximations converge almost surely in p-variation to Stratonovich Brownian motion, a geometric p-rough path.Brownian motion has finite p-variation for p in (2, 3), providing the rough-path setting used in the SDE treatment.Source: §C.3.3.1 · p. 179
For bounded-variation paths the extra term is determined by the base path, but Brownian paths require an integration choice, such as Stratonovich rather than Itô.Brownian paths are not of bounded variation, so integration is not unique; another convention, such as Itô, could also be chosen.Source: §C.3.3.1 · p. 179
Lifting a path into R × Rd × Rd×d defines p-variation convergence and integration against the lift, which is treated as fundamental while the base path is its projection.The lift fixes the integration convention, here Stratonovich.Source: §C.3.3.1 · p. 180

C.3.3.2 Rough differential equations and the universal limit theorem

The universal limit theorem defines rough differential equations through limits of bounded-variation-driven equations, and its Stratonovich corollary gives rough-path solutions driven by enhanced Brownian motion. The rough-path approach separates stochasticity, roughness, and control.

Rough differential equations and the universal limit theorem

Theorem C.35 gives a unique universal rough-path limit Y for y_n, independent of the approximating path sequence [LCL04, Theorem 5.3], [FV10, Theorems 10.29, 10.50, 10.57].It assumes p∈(2,3), γ>p, a linear or Lip(γ) vector field, convergent initial states, and bounded-variation drivers converging in p-variation to X; y_n converges to Y in p-variation.Source: §C.3.3.2 · p. 180
Under linear or Lip(γ) drift and diffusion, approximations driven by lifts converging to Stratonovich Brownian motion converge almost surely in p-variation to the unique rough-path solution of the Stratonovich SDE.The solution starts at Y(0)=(1, ζ, 0) and satisfies the equation with drift integrated against dT and diffusion against dW.Source: §C.3.3.2 · p. 181
The equation is driven by an enhanced path, so Stratonovich stochasticity is encoded in W and one can sample W before deterministically solving the RDE.The enhanced path includes the path and its second-level component.Source: §C.3.3.2 · p. 181
An RDE can be approximated by a sequence of CDEs, manipulated there, and then recovered by taking a limit.Source: §C.3.3.2 · p. 181
Roughness, stochasticity, and control are factored apart, unlike in the classical SDE approach, which combines these ideas.Source: §C.3.3.2 · p. 181

C.3.3.3 Rough adjoints … C.4.2 Stability

The appendix establishes a rough-path adjoint identity for Stratonovich SDEs and gives convergence and stability results for the reversible Heun method on ODEs.

C.3.3.3 Rough adjoints

The rough adjoint exists uniquely and, for almost all Brownian sample paths, its first component equals the derivative of the terminal loss.The proof obtains the adjoint as a rough differential equation; its non-rough equivalent is not technically a Stratonovich SDE because the adjoint is not measurable with respect to the natural filtration of the Brownian motion.Source: §C.3.3.3 · pp. 182, 184

C.3.4.1 On ODEs

The ODE optimise-then-discretise proof is new, simpler than most literature versions, and avoids meaningful reliance on sensitivity differentiability, enabling later CDE generalisation.The proof finds processes whose product has zero time derivative, using the interaction between forward and reverse sensitivities.Source: §C.3.4.1 · p. 185

C.3.4.2 On CDEs

The CDE optimise-then-discretise proof is new and substantially simpler than the unpublished proof in [Kid+20b, Appendix A.1].The earlier result had not been formally published.Source: §C.3.4.2 · p. 185

C.3.4.3 On SDEs

The SDE presentation follows the lines of the unpublished [Kid+20b, Appendix A.2, Appendix A.3], but simplifies it and fixes technical holes.The discussion notes that the treatment considers initial-condition derivatives, while derivatives with respect to the driving signal or vector field are also possible; to the authors’ knowledge, the literature lacks a complete account across these cases and ODEs, CDEs, SDEs, and general RDEs, though forward sensitivity results appear in [FV10, Theorem 4.4, Theorem 11.3, Exercise 11.10].Source: §C.3.4.3 · pp. 185–186

C.4. CONVERGENCE AND STABILITY OF THE REVERSIBLE HEUN METHOD

The SDE adjoint theorem assumes Lip(γ + 1) regularity for simplicity, and the text expects weaker drift and diffusion regularity may suffice.The authors suggest the drift may need only minimal regularity and that substituting another universal limit theorem could permit Lip(γ) regularity on the diffusion.Source: §C.3.4.3 · p. 186, §C.4 · p. 187

C.4 Convergence and stability of the reversible Heun method

The reversible Heun method is introduced through a two-step ODE update, while the proof leaves some steps implicit.The implicit details include incorporating y_n into the control when taking the limit in Φ_n and viewing the Jacobian as a solution map from t to T.Source: §C.4 · p. 187

C.4.1 Convergence

For ODEs, the reversible Heun method is second-order, equivalent in its two-step update to the leapfrog/midpoint method.The full SDE case is referred to Section 5.3.2.2 and [Kid+21a, Appendix D].Source: §C.4.1 · p. 187, §C.4 · p. 187

C.4.2 Stability

The reversible Heun method’s ODE stability region is [−i, i], with stability defined by boundedness of the numerical iterates.This boundedness criterion matches the analytical solution’s behavior for Re(λ) ≤ 0; some authors instead require numerical iterates to converge to zero.Source: §C.4.2 · p. 188

C.4. CONVERGENCE AND STABILITY OF THE REVERSIBLE HEUN METHOD … D.2 Latent ODEs on decaying oscillators

The appendices establish the reversible Heun method’s stability behavior, describe Brownian Interval sampling and its memory-efficient query strategy, and specify experimental setups for two examples. The supplied passages report no outcomes for those experiments.

C.4. CONVERGENCE AND STABILITY OF THE REVERSIBLE HEUN METHOD

Outside λ∆t ∈ [−i, i], |y_n| is unbounded, while within the stability region it is bounded; the region matches asynchronous leapfrog [Zhu+21] and leapfrog/midpoint [Sha09].The boundedness and instability claims follow from the behavior of the recurrence terms as n grows.Source: §C.4 · p. 189

C.5.2 Discussion

Brownian Interval sampling traverses a binary tree to partition a requested interval, then samples its increment using seeded bridge values; the appendix continues its definition and discusses technical considerations for Brownian sampling.A node stores an interval, a seed, and optional parent and child nodes; query traversal starts from a hint node for efficiency.Source: §C.5.1 · p. 190, §C.5 · p. 190, §C.5.2 · p. 190

C.5. BROWNIAN INTERVAL

Nearby solver queries let Brownian Interval search start from a recent-node hint, giving modal O(1) search; queries are usually over small intervals and return one or two nodes.The cited discussion says long queries such as [0, s] can return inefficiently long node lists, but SDE solvers generally do not make them.Source: §C.5 · p. 193
A prebuilt dyadic tree paired with an LRU cache avoids repeated root recomputation on backward queries, reducing recomputation to logarithmic time within cached subintervals.Without this structure, the backward pass can take O(n^2) time; the tree’s smallest intervals are chosen using an estimate of the solver’s average step size and cache size.Source: §C.5 · pp. 193–194
The Brownian Interval’s GPU memory use is constant because its LRU cache has fixed size, although its dependency tree grows in CPU memory; traversal may also need tail recursion or trampolining.Deep recursion can occur when a query crosses the midpoint of the prespecified tree.Source: §C.5 · p. 194

D.1 Continuous normalising flows on images

The continuous-normalizing-flow example uses fixed-step Tsitouras 5(4) integration with discretise-then-optimise backpropagation and exact trace-Jacobian calculations.The target, cat, and butterfly datasets use respectively two, two, and three stacked CNFs; the target hidden layers have width 128, while cat and butterfly use width 64.Source: §D.1 · pp. 195–196

D.2 Latent ODEs on decaying oscillators

The latent-ODE example uses 10,000 paths solving a linear differential equation, with d=2 observations at 20 random times; it trains over [0,T], T∼Uniform[2,3], tests over [0,12], and uses an autonomous 16-dimensional state.Initial states are independently sampled from a two-dimensional standard normal distribution. The solver is Dormand–Prince 5(4) with fixed timestep 0.4, and backpropagation uses discretise-then-optimise.Source: §D.2 · p. 196

D.3. NEURAL CDES ON SPIRALS … Index

This section presents appendix experiments and implementation details spanning neural CDE spiral classification, neural SDE examples, symbolic regression on a nonlinear oscillator, and a BIDMC neural RDE experiment. It also specifies the RDE model and training setup for comparison with a baseline neural CDE.

D.3 Neural CDEs on spirals

The spiral dataset contains 128 clockwise and 128 counterclockwise series, each with 100 regularly sampled points over [0, 4π].Initial points were sampled from the unit circle, and half the series were flipped in the y axis.Source: §D.3 · p. 197
The neural CDE was reduced to an ODE and solved with adaptive Tsitouras 5(4), using discretise-then-optimise backpropagation.Training used Hermite cubic spline interpolation, binary cross-entropy, and Adam for 20 steps with batch size 32.Source: §D.3 · pp. 197–198
100% (test) accuracy was achieved by the final neural CDE model.The experiment used 256 time series, split evenly between clockwise and counterclockwise spirals.Source: §D.3 · pp. 197–198

D.4.1 Brownian motion

For Brownian-motion generation, the neural SDE and CDE used reversible Heun with unit steps, while generator and discriminator training used simultaneous gradient descent.The dataset had 8192 samples of v + w(t), with 11 regularly sampled points per series; RMSprop training used batch size 1024 for 10,000 steps.Source: §D.4.1 · p. 198

D.4. NEURAL SDES ON TIME SERIES

The discriminator used a fivefold larger learning rate than the generator so it could keep pace and provide informative gradients.The passage explains that, with simultaneous gradient descent, a larger discriminator learning rate serves as an alternative to multiple discriminator updates per generator step.Source: §D.4 · p. 199

D.4.3 Damped harmonic oscillator

In the damped-oscillator example, the auxiliary network evaluates the observed trajectory at the nearest discrete timestamp within its input interval.The auxiliary network is parameterised as νφ(t, y, x) = νφ,1(t, y, νφ,2(x|[t,T])).Source: §D.4.3 · p. 200

D.4. NEURAL SDES ON TIME SERIES

For the Lorenz-attractor example, latent SDE training used Adam, while SDE-GAN training used Adadelta; the opposite optimizer choices appeared to fail.The appendix notes that the generator was associated with both optimizers in the SDE-GAN setup, while the latent SDE components used Adam and the discriminator used Adadelta.Source: §D.4 · p. 201

D.5 Symbolic regression on a nonlinear oscillator

O(10^-5) was the final L2 loss for the neural vector field in the nonlinear-oscillator symbolic-regression example.Regularised evolution searched equations on a loss-complexity Pareto front, and the best equation was selected by a criterion trading off loss against expression size.Source: §D.5 · p. 202, §D.6 · p. 203

D.6 Neural RDEs on BIDMC

At the time of the experiment, Diffrax had not yet been created and no JAX library for computing logsignatures was available.Source: §D.6 · p. 203
The dataset is split 70%/15%/15% for training, validation, and testing; each series has 4,000 points sampled at 125 Hz, with channels normalized to zero mean and unit variance.Source: §D.6 · p. 203
The RDE is reduced to an ODE and solved with fixed-step RK4 with 3/8 rule; linear interpolation is used for data and logsignatures, as data-space linear interpolation enables efficient logsignature computation.Source: §D.6 · pp. 203–204
Training uses L2 loss, batch size 512, and learning rate 6.25 × 10^-5, with learning-rate reduction after 15 validation-loss plateaus and early stopping after 60 epochs without improvement.Training rolls back to the checkpoint with the best validation loss.Source: §D.6 · pp. 203–204
Each network is an MLP with three 192-wide ReLU hidden layers and a 64-dimensional evolving state, selected through hyperparameter optimization for the competing neural CDE baseline.Source: §D.6 · p. 203