Source-linked AI summary
Neural SDEs as Infinite-Dimensional GANs
Patrick Kidger, James Foster, Xuechen Li, Harald Oberhauser, Terry Lyons
TL;DR
SDE models have traditionally been relatively inflexible and are often fitted through prespecified statistics rather than directly over generated paths. This paper couples a neural SDE generator with a neural CDE discriminator in a Wasserstein GAN, bringing neural and classical SDE modelling together. The authors state that arbitrary drift and diffusion functions are admissible and that arbitrary SDEs may be learned in the infinite-data limit, subject to solver-based training considerations.
Problem
Traditional SDE modelling is relatively inflexible and commonly fits parameters by matching prespecified statistics, while SDE densities are unavailable in the usual sense.
Method
The paper trains a neural SDE and neural CDE as generator and discriminator in a Wasserstein GAN, using sampled paths rather than density evaluation.
Results
The authors state that arbitrary drift and diffusion functions are admissible and that arbitrary SDEs may be learned in the infinite-data limit.
Takeaways & Limitations
Neural SDEs provide continuous-time generative models for distributions over path space while extending existing classical SDE modelling workflows.
Takeaways & Limitations
Gradient-penalty training requires double backpropagation, so continuous-time adjoint gradients can be inaccurate at moderate solver step sizes; the authors instead backpropagate through solver operations.
Abstract
from arXiv · showhide
Stochastic differential equations (SDEs) are a staple of mathematical modelling of temporal dynamics. However, a fundamental limitation has been that such models have typically been relatively inflexible, which recent work introducing Neural SDEs has sought to solve. Here, we show that the current classical approach to fitting SDEs may be approached as a special case of (Wasserstein) GANs, and in doing so the neural and classical regimes may be brought together. The input noise is Brownian motion, the output samples are time-evolving paths produced by a numerical solver, and by parameterising a discriminator as a Neural Controlled Differential Equation (CDE), we obtain Neural SDEs as (in modern machine learning parlance) continuous-time generative time series models. Unlike previous work on this problem, this is a direct extension of the classical approach without reference to either prespecified statistics or density functions. Arbitrary drift and diffusions are admissible, so as the Wasserstein loss has a unique global minima, in the infinite data limit any SDE may be learnt. Example code has been made available as part of the \texttt{torchsde} repository.
1. Introduction
The paper combines neural networks with SDEs to increase modelling capacity while retaining continuous-time stochastic dynamics. It reframes classical SDE fitting as a Wasserstein GAN using a neural SDE generator and neural CDE discriminator.
- 1.1. Neural differential equations: Neural differential equations parameterise differential-equation vector fields with neural networks, providing trainable high-capacity function approximators.This hybridisation combines neural-network and differential-equation modelling paradigms.
- 1.2. Stochastic differential equations: SDEs model continuous-time systems subject to uncertainty through deterministic and stochastic dynamics driven by Brownian motion.The state process, drift, diffusion, and Brownian motion define the stochastic evolution.
- 1.2. Stochastic differential equations: Classical SDE workflows formulate a model and calibrate its parameters by matching prespecified target statistics, such as financial payoff functions.A common modelling choice adds fixed diffusion noise to an existing ODE model.
- 1.3. Contributions: The paper generalises classical SDE fitting through a Wasserstein GAN that couples a neural SDE generator with a neural CDE discriminator.This approach learns statistics through the discriminator rather than relying on prespecified statistics.
- 1.3. Contributions: Arbitrary drift and diffusion functions are admissible, and the authors state that arbitrary SDEs may be learned in the infinite-data limit.This claim follows from the unique global minimum of the Wasserstein loss.
- 1.3. Contributions: The method is presented as a direct extension of classical SDE modelling that uses neither prespecified statistics nor density functions.Neural SDEs are thereby framed as continuous-time generative models for distributions over path space.
2. Related work
Prior neural SDE work spans terminal-state generation, latent-variable modelling, regularisation, and path-space modelling. The paper positions its approach as a continuous-time GAN that extends classical SDE workflows with a learned discriminator.
- Previous formulations: Earlier neural SDE formulations often use stochastic dynamics to inject noise while treating the terminal state as the quantity of interest.This includes variational, latent-variable, Feynman–Kac, and score-based approaches.
- Previous formulations: Several studies use stochasticity to enhance or regularise neural ODEs, with some reporting that the optimal diffusion becomes zero.This issue is noted for random-ODE limits and related neural SDE training settings.
- Previous formulations: Other work models full time-varying paths, including approaches based on non-characteristic maximum mean discrepancy objectives.These methods target distributions over path space rather than only terminal values.
- SDEs as GANs: The paper’s approach treats a neural SDE as a continuous-time model component and combines it with a neural CDE discriminator in a GAN framework.Both SDEs and GANs transform random noise into samples without requiring tractable densities.
3. Method
The method views SDEs as transformations from Brownian-motion noise to path distributions and combines a neural SDE generator with a neural CDE discriminator. Training matches generated and observed path statistics without requiring tractable probability densities.
- SDE–GAN connection: SDEs map Brownian-motion noise to a probability distribution on continuous paths, which can be sampled with numerical solvers.The strong solution defines the transformation, while sampling remains available even when ordinary probability densities are not.
- SDE–GAN connection: The GAN formulation trains a generator by matching statistics of generated samples with statistics of the target data.This avoids the need to evaluate the model’s probability density and uses a discriminator as a learnt statistic.
- Generator: The neural SDE generator uses Brownian motion and an additional initial noise variable, with neural networks parameterising its initial condition, drift, and diffusion.The hidden state X is read out to produce the output path Y, whose distribution is trained to approximate the target path distribution.
- Numerical solution: The framework admits either Stratonovich or Itô solutions because the vector fields are learnt, while the experiments use midpoint or Euler–Maruyama solvers respectively.The choice of solution convention is therefore not mandatory for the construction.
- Discriminator: The discriminator processes each generated or observed path with a neural CDE whose terminal hidden state is mapped to a scalar real-versus-fake score.Its hidden-state dimension h is a hyperparameter, and the formulation is motivated as a continuous-time analogue of recurrent neural networks.
- Data handling: Dense irregularly sampled observations are interpolated into continuous paths before being supplied to the discriminator.The data may contain missing values, provided the series is informally densely sampled.
4. Experiments
Experiments evaluate neural SDEs on synthetic, financial, air-quality, and SGD-weight datasets, comparing them with other neural differential-equation models. Across these settings, neural SDEs recover the synthetic process and achieve strong comparative performance, while training success depends on several practical choices.
- Datasets: Experiments cover four regimes: a univariate synthetic process, Google/Alphabet stocks, conditional Beijing air quality, and weights evolving under SGD.The stocks dataset contains approximately 14.6 million datapoints.
- Synthetic example: Neural SDEs accurately recover the true marginal distributions and sample paths of a time-dependent one-dimensional Ornstein–Uhlenbeck process.Marginals are evaluated at t = 6, 19, 32, 44, and 57, while sample-path comparisons use 50 samples from each distribution.
- Google/Alphabet stocks: On the stocks dataset, neural SDEs outperform both competing models on all reported metrics, while Latent ODE fails completely.The competitors are Latent ODE and the extended continuous time flow process (CTFP).
- Air quality: On Beijing air quality, neural SDEs win on prediction and MMD, whereas CTFPs lead classification but severely underperform on prediction.The task is conditional on class labels representing 14 measurement locations.
- SGD-trained weights: On the SGD-weights dataset, neural SDEs outperform CTFPs by an order of magnitude on prediction and by about twofold on MMD, despite CTFPs’ small classification edge.Latent ODEs perform relatively poorly compared with both alternatives.
- Training considerations: Successful neural-SDE training benefited from final tanh nonlinearities, stochastic weight averaging, nonzero weight decay, and Adadelta optimization.Adadelta performed substantially better than the other optimizers considered, although the authors provide no explanation.
5. Conclusion
The paper couples a neural SDE generator with a neural CDE discriminator to train neural SDEs as continuous-time GANs and model distributions over path space.
- A neural SDE and neural CDE form a generator/discriminator pair for continuous-time GAN training.
- The approach extends the classical SDE-modelling workflow using prespecified payoff functions.
- The demonstrated capability is modelling distributions over path space with neural SDEs.
A.1. General notes
The experiments use standardized neural-network architectures, numerical solvers, training procedures, and software components across the compared models and datasets.
- Solvers: SDEs and ODEs are solved with the midpoint method, while classification and prediction CDEs are reduced to ODEs.
- Architectures: The neural SDE uses neural-network parameterizations for its generator initial condition, drift, diffusion, and discriminator vector fields.
- Architectures: Stocks, air quality, and weights models use two-hidden-layer width-64 feedforward networks with softplus activations and final tanh nonlinearities.The final tanh operation was reported to improve performance for every model.
- Noise: Neural SDEs use normally distributed initial noise and Brownian-motion noise, with 40-dimensional initial conditions for the stocks, air quality, and weights datasets.The Brownian-motion dimension was dataset dependent.
- Training: Training uses 100 epochs for stocks, air quality, and weights, with five discriminator steps per neural-SDE generator step.
- Classifier and predictor: The classifier is a neural CDE, while the predictor is a neural CDE/neural ODE encoder/decoder pair, each using hidden state and hidden-layer widths of 32.
A.2. Time-dependent Ornstein–Uhlenbeck process
The synthetic experiment evaluates neural SDEs on a time-dependent one-dimensional Ornstein–Uhlenbeck process using fixed sample, batch, noise, and network settings.
- Training setup: Each synthetic sample has length 64, with batch size 1024 and neural-SDE training for 6000 steps.L2 weight decay of 0.01 and weight averaging over the final 5500 steps were also used.
- Architecture and noise: The Brownian-motion noise has 3 dimensions, the initial noise has 5 dimensions, and evolving hidden states have size 32.The MLP width is 16 with a single hidden layer.
A.3. Stocks
The stocks experiment uses length-100 samples, batch size 2048, three-dimensional Brownian noise, and a prediction metric based on forecasting the final 20% from the first 80%.
- Data and batching: Each stock-price sample has length 100, and every model uses batch size 2048.
- Neural-SDE training: The neural SDE uses three-dimensional Brownian motion and weight averaging over every training epoch.Its discriminator receives one preliminary training epoch before joint generator-discriminator training.
- Prediction: The prediction metric forecasts the last 20% of each input using its first 80%.
A.4. Beijing Air Quality
The Beijing air-quality experiment used standardized, fixed-length samples and pretrained the neural SDE discriminator before joint training. Its prediction metric used an equal 50%/50% history-to-future split.
- Each sample was length 24, normalized to zero mean and unit variance, with batch size 1024 for every model.
- The neural SDE discriminator received 10 preliminary training epochs before generator–discriminator training commenced.
- Weight averaging for the neural SDE covered the final 40 training epochs for both generator and discriminator.
- The prediction metric used the first 50% of each input to predict its last 50%.
A.5. Weights
The weights experiment modeled trajectories of individual convolutional-network weights during MNIST training. It compared neural SDE and latent ODE models using larger batches than the CTFP and an 80%/20% prediction split.
- Each sample was a length-100 trajectory for one scalar weight tracked epoch-by-epoch during 50-epoch MNIST training.Every network weight became a separate sample, repeated across 10 training runs.
- The resulting dataset contained 10P samples of length 100 with one channel, where P is the convolutional network’s parameter count.
- The neural SDE and latent ODE used batch size 4096, while the memory-intensive CTFP used batch size 1024.
- The neural SDE discriminator received 10 preliminary training epochs before joint generator–discriminator training began.
- The prediction metric used the first 80% of each input to predict the last 20%.