PaperlayerView exact revision on arXiv

Source-linked AI summary

On Neural Differential Equations

Patrick Kidger

arXiv:2202.02435v1cs.LGmath.CAmath.DSmath.NAstat.ML

TL;DR

This thesis surveys neural differential equations and their applications across mathematical modelling and machine learning. It discusses practical methods and reports benefits including black-box solver use, adaptive-solver advantages, and efficient SDE queries, while noting that continuous-time models may offer little for image classification.

  • Problem

    Symbolic regression involves nondifferentiable expression spaces and a very large number of candidate expressions, while recurrent-network training can break down for increasingly long time series.

  • Method

    The thesis provides an accessible survey of neural differential equations, including hybrid modelling in which neural networks approximate theoretically undescribed differential-equation terms.

  • Results

    The survey covers applications and methods including black-box differential-equation solvers, checkpointing, adaptive solvers, and efficient cached queries for stochastic differential-equation solvers.

  • Takeaways & Limitations

    Neural differential equations can support applications in mathematical modelling and machine learning while accommodating practical solver and irregular-time-series challenges.

  • Takeaways & Limitations

    For image classification, continuous-time models usually offer little benefit, and traditional explicitly discretised residual networks are easier to work with.

Abstract

from arXiv · show

The conjoining of dynamical systems and deep learning has become a topic of great interest. In particular, neural differential equations (NDEs) demonstrate that neural networks and differential equation are two sides of the same coin. Traditional parameterised differential equations are a special case. Many popular neural network architectures, such as residual networks and recurrent networks, are discretisations. NDEs are suitable for tackling generative problems, dynamical systems, and time series (particularly in physics, finance, ...) and are thus of interest to both modern machine learning and traditional mathematical modelling. NDEs offer high-capacity function approximation, strong priors on model space, the ability to handle irregular data, memory efficiency, and a wealth of available theory on both sides. This doctoral thesis provides an in-depth survey of the field. Topics include: neural ordinary differential equations (e.g. for hybrid neural/mechanistic modelling of physical systems); neural controlled differential equations (e.g. for learning functions of irregular time series); and neural stochastic differential equations (e.g. to produce generative models capable of representing complex stochastic dynamics, or sampling from complex high-dimensional distributions). Further topics include: numerical methods for NDEs (e.g. reversible differential equations solvers, backpropagation through differential equations, Brownian reconstruction); symbolic regression for dynamical systems (e.g. via regularised evolution); and deep implicit models (e.g. deep equilibrium models, differentiable optimisation). We anticipate this thesis will be of interest to anyone interested in the marriage of deep learning with dynamical systems, and hope it will provide a useful reference for the current state of the art.

Originality

The thesis presents original research alongside a survey of neural differential equations, with substantial contributions to software, papers, and additional unpublished material.

  • The thesis combines original collaborative research with a survey of neural differential equations.
  • A substantial contribution was the development of open-source differential-equation software for JAX and PyTorch.The libraries include solvers for ordinary, controlled, and stochastic differential equations.
  • The thesis includes research on neural controlled differential equations, neural SDEs, efficient gradients, online prediction, and long time series.
  • Additional unpublished material covers symbolic regression, universal approximation, neural-differential-equation parameterisations, and differential-equation sensitivities.
  • Other PhD papers and software libraries were developed but explicitly excluded from this thesis.

Introduction

The thesis introduces neural differential equations as a meeting point between differential-equation modelling and deep learning, and surveys their structure, connections, advantages, and applications.

  • 1.1 Motivation: The thesis aims to describe original research and provide an accessible survey of the rapidly developing field of neural differential equations.It presents the survey as, to the authors’ knowledge, the first written survey on the topic.
  • What is a neural differential equation anyway?: A neural differential equation uses a neural network to parameterise a differential equation’s vector field.The resulting differential equation is part of a differentiable computation graph that can be trained.
  • What is a neural differential equation anyway?: Neural ODE classification evolves an input from an initial condition, applies an affine transformation, and then uses a softmax to produce class probabilities.
  • The case for neural differential equations: The introduction connects neural differential equations with recurrent architectures, stochastic models, generative modelling, and classical mathematical modelling.
  • Connections to deep learning: Neural ODEs connect to residual networks because explicit Euler discretisation recovers the residual-network formulation.
  • The case for neural differential equations: Neural differential equations offer high-capacity approximation and a coherent theory of model quality, while treating discrete data in continuous time can accommodate irregular data.

Neural Ordinary Differential Equations

Neural ODEs parameterise differential-equation vector fields with neural networks, connecting continuous dynamical systems to familiar neural architectures. They support mechanistic augmentation, generative modelling, and efficient computation, while requiring numerical solution and differentiation through the equation.

  • Neural ODE definition: A neural ODE uses a neural network fθ to parameterise a differential equation’s vector field and evolve an any-dimensional tensor y0.The network is typically a feedforward or convolutional architecture, and θ contains learned parameters.
  • Well-posedness: Lipschitz continuity of the neural vector field ensures existence and uniqueness of the ODE solution through Picard’s theorem.The relevant Lipschitz constant is independent of time.
  • Computation: Neural ODE models require both numerical solution of the differential equation and backpropagation through it to obtain parameter gradients.Analytic solutions are generally unavailable, while software for these tasks is standardised.
  • Image classification: Continuous-time models are often unnecessary for image classification, where explicitly discretised neural ODEs—traditional residual networks—are easier to work with.The thesis presents image classification as an example rather than recommending neural ODEs for this task.
  • Universal differential equations: For hybrid physical modelling, neural networks can represent residual behaviour between theoretical predictions and observations without changing the fundamental fitting procedure.This approach is called a universal differential equation and can encode known problem structure as an inductive bias.
  • Generative modelling: Continuous normalising flows can learn complex two-dimensional distributions with multiple modes and fine-scale filaments.Their divergence integrand can be estimated in O(d) work, reducing general normalising-flow log-determinant-Jacobian cost from O(d3) to O(d2) or O(d).

4 (Neural Stochastic Differential Equations)

This section develops latent ODE models for generative time series, including their decoder construction, training, sampling, irregular-time outputs, and extrapolation. It also discusses neural ODE expressivity, memory efficiency, and limitations arising from topology preservation and restrictive assumptions.

  • Latent ODE construction: A latent ODE maps a latent sample through a neural decoder into an evolving latent state whose time-indexed distributions form the model output.The decoder initializes y(0)=gθ(z), evolves y by a neural ODE, and uses the state to parameterize a probability density.
  • Training and sampling: Training uses the standard VAE objective: reconstruction encourages replication of inputs, while regularization matches the latent initial distribution to a samplable known distribution.The encoder produces a diagonal-covariance multivariate normal distribution, and the objective is optimized end to end.
  • Training and sampling: Sampling draws z from a standard normal, decodes it to y0, solves the neural ODE forward, and returns the resulting time-indexed distribution or a point statistic.Gaussian or Laplace output distributions suit point statistics, while more expressive distributions can support uncertainty quantification.
  • Irregular sampling and extrapolation: A latent ODE produces excellent samples over [0, 12], despite training on an interval four times shorter, demonstrating good extrapolation on decaying oscillators.The model is evaluated over a continuous-time output interval, unlike an analogous RNN restricted to prespecified discrete timestamps.
  • Numerical and architectural properties: Continuous-time models handle irregular sampling, while ODE theory also motivates residual-network variants with favorable properties and reversible recomputation that improves memory efficiency.Recomputing intermediate states during backpropagation avoids storing the corresponding residual-network activations, at the cost of extra computation.
  • Expressivity and limitations: Neural ODE expressivity is constrained because continuous flows preserve input topology and cannot make trajectories cross, preventing some linear separations without augmentation.Universal approximation can instead be obtained through a universal vector field or through increased latent dimensionality with affine maps, and the latter result is presented as new here.

Neural Controlled Differential Equations

Neural controlled differential equations extend differential equations to sequential data by incorporating incoming observations through a control path. They provide continuous hidden-state updates, universal approximation under a time-augmented formulation, and practical links to recurrent networks and irregular observations.

  • Motivation: CDEs address the ODE limitation that later-arriving observations cannot directly alter a solution determined by its initial condition.They incorporate incoming information through controlled differential equations driven by an input path.
  • CDE formulation: A controlled differential equation maps an input path x to an output path y through a vector field and Riemann–Stieltjes integration.The control path is assumed continuous and of bounded variation, while the vector field is Lipschitz for the stated existence-and-uniqueness result.
  • CDE formulation: CDEs can compute functions of their controls, including both the value and first integral of an arbitrary input signal.The example drives a two-dimensional state with t ↦ (t, x(t)); the construction is not specific to sin(t).
  • Neural CDE model: The neural CDE prediction updates continuously as input arrives and converges to the correct steady-state classification on the described spiral task, achieving perfect accuracy.The model is trained with binary cross entropy on the terminal sigmoid-affine output.
  • Theory: With the extra time-like variable included, an affine map of the terminal hidden state is a universal approximator from sequences in R^dx to R.The thesis states this as an informal universal-approximation theorem for neural CDEs.
  • Comparison with RNNs: Neural CDEs have continuous dependence on observed data, whereas recurrent models modify their hidden state at observations and may evolve it between observations.Their computational work scales with gaps between observations, and their memory can be O(H + T) using optimise-then-discretise backpropagation.
  • Comparison of formulations: The CDE form fθ(y(s)) dx(s) can represent the identity between paths, while the alternative fθ(y(s), x(s)) form cannot; the converse expressivity statement is not true.This distinction motivates the theoretically preferred control-path formulation and connects it to integration against Brownian motion.

Neural Stochastic Differential Equations

Neural stochastic differential equations extend continuous-time stochastic modelling with learned dynamics, supporting generative time-series and static distribution modelling. They retain stochastic structure while connecting naturally to recurrent architectures and offering memory-efficient training and broad approximation capacity.

  • Foundations: SDEs model continuous-time systems evolving under uncertainty through deterministic and stochastic terms, with solutions guaranteed under mild regularity conditions.The drift and diffusion functions are assumed Lipschitz, with finite second moment at initialization.
  • Calibration: Classical SDE calibration matches expectations of prespecified witness or payoff functions between generated and observed data.The model depends implicitly on parameters θ, which are calibrated from real-world data.
  • Neural SDEs: Neural SDEs replace simple theoretical constructions with learned vector fields while preserving inherent randomness and generative modelling capabilities.A fixed diffusion matrix can be added to an existing ODE, whereas neural SDEs learn more flexible dynamics.
  • Neural SDEs: A discretised neural SDE corresponds to an RNN whose input is Brownian noise and whose output is a generated sample.Euler–Maruyama uses Gaussian increments Δw_j ∼ N(0, Δt) at fixed time steps.
  • Generative modelling: Neural SDEs represent both continuous-time generated paths and terminal-value distributions, paralleling continuous normalising flows with Brownian rather than initial-condition randomness.The terminal value y(T) is a sample from a distribution over R^dy.
  • Properties: Training can reduce memory asymptotics from O(H + T) to O(H), while suitable neural vector fields can approximate any Markov SDE.The solution and discriminator output are not recorded during training, and approximation follows from universal neural-network and SDE results.
  • Properties: The recurrent discriminator has Lipschitz constant O(λT), requiring specialized regularisation; careful clipping is preferred over a naive gradient-penalty approach.The gradient-penalty alternative works but mostly has disadvantages compared with careful clipping, and its derivative is defined through a discretised solution.
  • Examples: In examples, neural SDEs match chaotic multidimensional time-series distributions and handle processes with both drift and diffusion that are difficult for pure-drift or nearly pure-diffusion models.The Brownian-motion example matched drift well but retained excessive diffusion, showing that additional training may be needed.

Numerical Solutions of Neural Differential Equations

The thesis compares approaches for solving and differentiating through neural differential equations, emphasizing trade-offs among memory, gradient accuracy, computational cost, and solver compatibility. It also develops reversible solvers, Brownian reconstruction methods, and hypersolvers for efficient numerical computation.

  • Adjoint methods: The continuous adjoint method reduces memory from O(HT) to O(H), independent of the time horizon T.It recomputes the forward solution during the backward pass instead of storing all internal solver operations.
  • Adjoint methods: Continuous adjoints can treat the differential equation solver as a black box, unlike discretise-then-optimise implementations requiring autodifferentiable solvers.This permits reuse of solvers written for other purposes or software packages.
  • Adjoint methods: Continuous adjoints compute less accurate gradients than backpropagation through the solver and may slow training, affect final performance, or cause failure.The additional numerical solution of the adjoint equations introduces further numerical error.
  • Reversible solvers: Reversible differential equation solvers combine memory efficiency with accurate gradients, while requiring extra computation to reconstruct the forward solution.When the forward solution is recovered, the gradients match discretise-then-optimise gradients for the numerical discretisation.
  • Efficient solvers: Hypersolvers can replace 80 Dormand–Prince steps with 2 HyperHeun steps without degrading accuracy in a reported continuous normalising flow example.The approach uses a simple base update rule and a learned correction model.
  • Brownian reconstruction: The Brownian Interval provides exact Brownian-motion sampling with O(1) query times, using a query-dependent interval tree and caching.Its worst-case query complexity is O(log(1/h)), while the Brownian Tree approach uses O(dwT) memory.

Miscellanea

This section presents symbolic regression for dynamical systems, combining neural ODE learning with symbolic recovery, and illustrates the approach on a nonlinear oscillator.

  • Symbolic regression: Symbolic regression derives shallow expressions from data, but its nondifferentiable and combinatorially large search space makes complex problems difficult.The number of binary trees grows rapidly with expression size, and the task often becomes intractable beyond simple problems.
  • Symbolic regression: SINDy assumes paired observations of y and dy/dt and restricts f to a linear combination of preselected features.It uses an L1 sparsity penalty to select a small number of terms.
  • Neural regression: NDEs remove these assumptions by learning fθ from observed trajectories without requiring paired derivative observations, including when observations are sparse.The learned vector field can then be symbolically regressed against the observed states.
  • Limitations: In latent-space settings, symbolic regression is nontrivial because vector fields are identifiable only up to conjugacy.The Markov formulation assumes the vector field depends entirely on the observed state and not on the past.
  • Example 6.1: For the nonlinear oscillator, a neural ODE trained with an L2 loss reconstructs the observed trajectories and is reported to have perfectly learned the problem structure.The learned fθ is subsequently treated in isolation for symbolic regression via regularised evolution.
  • Example 6.1: Rounding each recovered constant to the nearest multiple of 0.01 produces the desired vector field.The symbolic-regression material is identified as new thesis material and joint work with Miles Cranmer.

Conclusion

The conclusion identifies neural differential equations as a rapidly developing field whose practical methods, reusable models, and connections to adjacent areas remain incomplete. It highlights neural PDEs, applications, and improved numerical methods as major directions.

  • Boutique versus off the shelf: Most neural differential equation applications remain boutique, with models tailored and retrained from scratch for individual use cases.The thesis contrasts this with traditional differential equations, which have numerous well-studied models and associated theory.
  • Neural ODEs: Neural ODEs offer an opportunity to extend tasks currently addressed only with non-neural ODEs across science, finance, and economics.The passage frames this as a significant opportunity rather than an established result.
  • Neural CDEs and SDEs: Neural CDEs and SDEs still require practical advances in expressive vector fields, efficient training, applications, and hybridisation with non-neural models.Connections to control theory, reinforcement learning, and several generative-modeling areas are described as largely unexplored or in their infancy.
  • Numerical methods: Numerical methods are presented as a major opportunity, including higher-order reversible ODE solvers and applying hypersolvers during training.The thesis emphasizes numerical methods because reversible solvers and hypersolvers remain underexploited.
  • Symbolic regression: Symbolic regression for dynamical systems remains underdeveloped: SINDy and its variants are established, while regularised evolution and deep symbolic regression are largely open areas.The thesis characterizes current symbolic-regression practice as more alchemy than science.
  • Neural PDEs: Neural PDEs are conspicuously absent from the thesis, and their methods have yet to converge because many PDE types must be considered.The conclusion calls neural PDEs a major open direction for the field.

A.1 Autodifferentiation

This appendix explains autodifferentiation modes, their memory and computational trade-offs, and related foundations for neural models handling transformations and irregular time series. It also summarizes signature-based CDE approximations.

  • Autodifferentiation: Autodifferentiation computes derivatives of compositions of differentiable primitives through the chain rule, including computation graphs represented as directed acyclic graphs.Frameworks automate these calculations using primitives such as matrix multiplication, trigonometric functions, and ReLUs.
  • Forward and reverse mode: Forward-mode autodifferentiation propagates sensitivities during the forward evaluation, whereas reverse-mode autodifferentiation computes them after the forward pass.Reverse mode must retain intermediate values, creating a potential memory concern.
  • Efficiency: For vector inputs and scalar outputs, reverse mode uses cheaper vector-matrix products than forward mode’s matrix-matrix products, explaining backpropagation’s common use.The appendix gives upper bounds of 2.5 dinput and 4 doutput times the cost of evaluating the function for forward and reverse mode, respectively.
  • Change of variables: Normalising-flow training is computationally expensive because evaluating the log-determinant Jacobian generally costs O(d3), motivating structured yet expressive architectures.After training, samples are generated by sampling X and evaluating f(X).
  • Universal approximation: Universal approximation means density in a function space; feedforward networks with nonpolynomial continuous activations densely approximate continuous functions on compact domains.The appendix also states a deep-and-narrow universal approximation theorem under nonaffine differentiability conditions.
  • Irregular time series: Irregular time series involve differing timestamps, variable lengths, and missing observations, requiring choices such as concatenating timestamps with values before modeling.The appendix describes missing entries using the symbol ∗.
  • Signatures: Logsignatures encode coordinate changes, Lévy-area information, and higher-order event ordering, allowing signature truncations to approximate CDE solutions over small timescales.The appendix presents this as information most important to solving the CDE.

B.1.2 The log-ODE method

The log-ODE method approximates a controlled differential equation by an ordinary differential equation whose vector field is constructed from truncated logsignature information. The approximation converges as the truncation level increases.

  • The log-ODE method: For a Lipschitz vector field and bounded-variation control, the log-ODE method constructs an ODE whose terminal solution converges to the CDE solution as M increases.The constructed vector field maps the hidden state into a space whose dimension depends on the truncated signature level.
  • The log-ODE method: The log-ODE right-hand side is a matrix-vector product between the constructed vector field and the truncated logsignature representation.The exact constructed vector field is known but expensive to compute, motivating an alternative treatment later in the section.
  • The log-ODE method: At truncation level M = 1, the constructed vector field equals f, and for small intervals the resulting equations give a first-order approximation to the CDE.With piecewise-linear control, this depth-1 approximation is exact.
  • Neural CDE setup: Neural CDEs use a continuous piecewise-linear interpolation of potentially irregular observations, with neural networks defining the vector field and initial hidden state.The CDE is typically solved by reducing it to an ODE.

B.2.1 Applying the log-ODE method

The log-ODE method compresses long input paths into slowly varying logsignatures, allowing neural CDEs to use larger integration steps and reduce computational and memory costs.

  • Method setup: The method partitions the input using m ≪ n points and represents each interval with a logsignature of depth M ≥ 1.The partition points and logsignature depth are hyperparameters.
  • Core mechanism: Logsignatures form a more slowly varying path than the original data, so the resulting differential equation can be solved with larger integration steps.This improves numerical behaviour relative to directly resolving the rapidly varying input path.
  • Efficiency: Typical speed-ups are about tenfold because the compressed path permits larger solver steps.Increasing step size accelerates training but makes updates less informative, whereas increasing depth slows training while incorporating more local information.
  • Implementation: The method preprocesses windowed logsignatures, interpolates them piecewise linearly, and then solves a neural CDE in the usual way.The required operations are available through existing software libraries.
  • Implementation: Directly modelling the lifted vector field avoids the computational expense of first modelling a lower-dimensional field and then computing its lifted form.The direct model maps R^dy to R^{dy×β(dx,N)}.
  • Trade-offs: The log-ODE method reduces memory from O(H + n) to O(H + m), but introduces depth and step-size hyperparameters and becomes less feasible as input channels grow.The number of logsignature channels β(d, M) grows exponentially in the number of input channels d.

B.2.4 Limitations

Experiments on long vital-sign time series show strong performance from depth-3 neural RDEs, while the method has tuning and input-dimensionality limitations.

  • Limitations: Using the log-ODE method requires tuning truncation depth and step size, which substantially affect training time and memory usage.In practice, both quantities are treated as modelling hyperparameters.
  • Limitations: The method is most feasible with few input channels because the number of logsignature channels grows exponentially in channel count.For larger input dimension, available parallelism may become saturated.
  • Results: The depth-3 neural RDE is the top performer for every task and step size, reducing test loss by 30–59% versus the corresponding neural CDE.The tasks predict respiratory rate, heart rate, or oxygen saturation from PPG and ECG data.
  • Results: The depth-3 neural RDE uses roughly an order of magnitude less training time, whereas the ODE-RNN baseline performs poorly and requires significantly more memory.The experiments use three BIDMC vital-sign prediction datasets and report L2 test loss, memory, and training time.
  • Interpretation: The authors attribute improved test loss to better learning of long-term dependencies after sequence-length reduction, with rough-model performance improving as step size increases.Increasing step size reduces the number of operations but retains less local information.

C.2.1.1 Universal approximation with respect to paths

Neural CDEs are universal approximators for continuous path functionals under compactness and uniqueness-of-signatures conditions, and the result extends to irregular time series through suitable interpolation.

  • Signature transform: The signature transform collects iterated Riemann–Stieltjes integrals up to a chosen depth, including the conventionally added leading scalar 1.These components encode path information at increasing interaction orders.
  • Path information: Including time as a channel allows path increments to be recovered from the signature, supporting the treatment of paths with their initial values represented explicitly.The increment x(s) − x(0) is determined by the signature of the augmented path.
  • Universal nonlinearity: Under compactness and uniqueness of signatures, linear functionals of signatures are dense in the continuous functions on path space.The signature therefore supplies a basis-like family for continuous path functionals.
  • Universal approximation: Neural CDEs inherit universal approximation with respect to controlling paths on compact path sets having uniqueness of signatures.The theorem establishes density in C(K; R^do).
  • Time-series extension: For discretised irregular time series, bounded interpolation and bounded variation yield relative compactness of the interpolated paths.The interpolation may include channels encoding observation counts and missingness.
  • Time-series extension: With the required interpolation conditions, neural CDEs are dense in continuous functions on the resulting time-series space.The construction is intended to be independent of the particular interpolation choice, provided its conditions hold.
  • Scope: The neural CDE setup directly subsumes models depending directly on the input path, so augmenting the state does not provide every possible advantage.The text notes that something is lost in the alternative construction considered.

C.3.3.2 Rough differential equations and the universal limit theorem

Rough differential equations define limits of differential equations driven by convergent enhanced paths, including Stratonovich Brownian motion, under suitable regularity assumptions.

  • Universal limit theorem: If vector fields have regularity Lip(γ) with γ > p and the driving paths converge in p-variation, the corresponding solutions converge to a unique geometric p-rough path.The universal limit depends only on the limiting path, vector field, and initial condition, not on the approximating sequence.
  • Stratonovich SDEs: The Stratonovich SDE case is obtained by applying the universal limit theorem to drift, diffusion, and Brownian-motion data.The resulting rough equation is driven by the enhanced Brownian motion.
  • Rough differential equations: A rough differential equation replaces integration against a continuous driving path with integration against an enhanced path containing first- and second-order information.The enhanced path lies in R × R^d × R^{d×d}.
  • Interpretation: The rough-path approach factors stochasticity, roughness, and control apart, unlike the classical SDE formulation described here.Rough equations can be approximated by CDEs, manipulated, and then passed to the limit.
  • Assumptions: The theory requires relatively high vector-field regularity to offset the roughness of the driving signal.The stated assumption is Lip(γ) regularity with γ > p.

C.3.3.3 Rough adjoints

The section derives rough adjoints for SDEs by interpreting the forward and reverse processes as rough differential equations and passing discretised approximations to the limit. It presents simpler, newly stated optimise-then-discretise proofs for ODEs, CDEs, and SDEs.

  • Rough adjoints: The adjoint process A is defined by a backwards-in-time linear rough differential equation coupled to the forward solution y.Its initial condition is set using the terminal derivative dL(y(T))/dy(T).
  • Proof strategy: The proof approximates Brownian motion with paths converging in p-variation, then applies the universal limit theorem to obtain rough-path limits.The construction augments states and controls so vector fields depend only on the state.
  • Proof strategy: The limiting adjoint satisfies a rough differential equation whose first projection is the desired process a, with a(t) = dL(y(T))/dy(t) for almost all sample paths.Uniqueness of solutions identifies the projected limiting process with the forward sensitivity equation.
  • Scope: The method extends directly to any rough differential equation driven by a geometric p-rough path.The extension is stated as a remark following the proof.
  • Novelty: The ODE proof is new and simpler than most existing versions because it does not meaningfully rely on differentiability of forward or reverse sensitivities.This feature enables the later generalisation to controlled differential equations.
  • Novelty: The CDE and SDE proofs are also presented as new, simpler proofs, while prior CDE and SDE proofs are identified in the literature or unpublished appendices.The section distinguishes published, unpublished, and newly developed proof sources.

C.4. CONVERGENCE AND STABILITY OF THE REVERSIBLE HEUN METHOD

This section establishes second-order accuracy and characterises stability for the reversible Heun method on ODEs, then discusses Brownian-interval data structures and their query complexity for SDE solvers.

  • Convergence and stability: When λ∆t ∈[−i, i], the numerical solution remains bounded independently of n; outside this interval, one component grows without bound.The conclusion follows from the behaviour of the characteristic quantities η and κ.
  • Convergence and stability: The reversible Heun method has stability region [−i, i] for ODEs.This is also the stability region of asynchronous leapfrog and leapfrog/midpoint methods.
  • Brownian Interval: The Brownian Interval samples increments by traversing a binary tree whose nodes partition the requested interval.The algorithm returns a sampled increment W_s,t and typically visits one or two nodes for solver-sized intervals.
  • Brownian Interval: Search hints exploit nearby forward and backward queries, giving modal search complexity O(1) rather than O(log(1/h)) in the average step size h.An LRU cache benefits because successive queries often share recently computed parent nodes.
  • Brownian Interval: A forward-built imbalanced tree can make the backward pass take O(n^2) time when cache misses trigger recomputation to the root.Snapshotting extra nodes is proposed as a solution, though it requires additional cache-engineering effort.

D.3. NEURAL CDES ON SPIRALS

The spiral experiment trains a neural CDE on regularly sampled clockwise and counterclockwise trajectories. The model achieves perfect test accuracy on this dataset.

  • Dataset: The dataset contains 256 time series, with 128 clockwise and 128 counterclockwise spirals generated from uniformly sampled initial angles.Each series has 100 regularly sampled points over [0, 4π].
  • Method: The neural CDE is reduced to an ODE and solved with the adaptive Tsitouras 5(4) solver.Backpropagation uses discretise-then-optimise.
  • Model: The model uses an MLP initial network and an MLP vector field, with an eight-dimensional evolving hidden state and sigmoid output.The vector-field MLP uses softplus activations and its output is passed through tanh.
  • Training: The loss function is binary cross-entropy, and training uses Adam with batch size 32 for 20 steps.Each step takes about 1.5 seconds on an A100 GPU.
  • Result: 100% test accuracy is achieved by the final model.The prediction updates toward the correct classification as the input sequence is processed.
Loading 2202.02435v1…