Modeling Temporal Data as Continuous Functions with Stochastic Process Diffusion

1. Introduction

• Time series are treated as discretized measurements of an underlying continuous function, with potentially arbitrary, nonuniform sampling.The motivation includes systems such as temperature or load, whose values exist and evolve smoothly over time.Source: §1 · p. 1
• Capturing the generative process of irregular data remains difficult, particularly because the data are inherently stochastic.Prior approaches include neural ordinary and stochastic differential equations, neural processes, and normalizing flows.Source: §1 · p. 1
• The method diffuses the whole underlying function with stochastic-process noise, preserving continuity, then learns to reverse the process to generate function samples.The illustrated approach also supports forecasting with uncertainty.Source: §1 · p. 1, Figure 1 · p. 2
• The paper uses conditioning for multivariate probabilistic forecasting and views its function-distribution model as a neural process for estimating missing points.The authors report that their model outperforms baselines on all tasks.Source: §1 · p. 1

2. Background

• Generative modeling learns a data distribution and generates samples by reversing a fixed process that adds noise.The section introduces discrete noise scales and their continuous-time SDE limit.Source: §2 · p. 2

2.1. Fixed-step diffusion

• Fixed-step DDPM adds Gaussian noise at increasing variance scales until the output is pure noise, then trains a model to reverse the process.The practical formulation predicts the added noise, and generation iteratively denoises a sample from the final distribution.Source: §2.1 · p. 2

2.2. Score-based SDE diffusion

• Score-based SDE diffusion replaces discrete steps with continuous evolution and learns the score function to solve the reverse SDE from noise to data.For the stated parameterization, the final noise is unit normal and the transition probability and score can be calculated in closed form.Source: §2.2 · pp. 2–3

2.3. Extensions

• Diffusion extensions target faster sampling or improved noise scheduling, and the paper describes these advances as orthogonal to its method.The authors say such techniques can be implemented to improve speed or sampling quality.Source: §2.3 · p. 3

3. Diffusion for time series data

• Time series are treated as observations of underlying continuous functions, so independent pointwise noise is unsuitable when it produces discontinuous samples.The formulation covers equally spaced and irregularly sampled observations.Source: §3 · p. 3

3.1. Stochastic processes as noise sources for diffusion

• Stationary GP and OU noise provide correlated, continuous noise with time-invariant variance; Wiener noise instead has growing variance and a degenerate value at time zero.The GP option uses an RBF kernel, while the OU process is a stationary alternative; both reduce to ordinary diffusion for a one-point series.Source: §3.1 · pp. 3–4

3.2. Discrete stochastic process diffusion (DSPD)

• DSPD replaces independent Gaussian noise with GP- or OU-process noise while retaining a closed-form noise-prediction loss and process-based sampling.The reverse model predicts noise for the full timestamped series, and the loss can be reparameterized so covariance Σ disappears.Source: §3.2 · p. 4

3.3. Continuous stochastic process diffusion (CSPD)

• CSPD extends continuous diffusion to GP- or OU-correlated noise, using a closed-form score and a network that predicts noise from the series, observation times, and diffusion time.The noise prediction is reparameterized for training, while the score is used when sampling.Source: §3.3 · pp. 4–5

3.4. Related work

• Compared with related function-diffusion approaches, the paper emphasizes conditioning on observed data for irregular time series and performing imputation.The cited alternatives use a Gaussian measure on Hilbert spaces, view function diffusion as neural processes, or approximate data in spectral space with KL decomposition.Source: §3.4 · p. 5

4. Applications

• Conditioning the reverse diffusion on past observations defines a probabilistic forecaster, while conditioning on observed values supports neural-process and imputation settings.The model predicts noise added to the clean series and can incorporate covariates at each time point.Source: §4 · p. 5

4.1. Forecasting multivariate time series

• Unlike the cited diffusion forecaster's discrete autoregressive predictions, this framework predicts at any future time point and produces multiple values in parallel.The paper states that parallel prediction scales better on modern hardware.Source: §4.1 · p. 5
• The forecasting model encodes history as z and uses noisy future values, diffusion steps, and future timestamps to predict noise with a two-dimensional convolutional architecture.The convolution's height and width correspond to feature and time dimensions.Source: §4.1 · pp. 5–6, Figure 2 · p. 6

1. Encode the history XH with an RNN to get z,

• The forecasting procedure encodes history XH with an RNN to obtain z, samples an initial future prediction from a GP prior, and then denoises it.The RNN is retained for similarity to the compared architecture, although transformers are also possible.Source: §1 · p. 6

4.2. Diffusion process as a neural process

• The neural-process version uses a deterministic encoder for observed points and conditions diffusion on z to sample values at arbitrary query times, directly modeling p(XB|XA).The encoder is permutation invariant, and the diffusion model learns the distribution of unobserved values given observations.Source: §4.2 · p. 6
• The model can interpolate or impute missing values across time by conditioning on observed points, with the stated appeal of representing different stochastic processes in one model.The paper connects this setup to a neural process, whose output distribution over functions depends on the input dataset.Source: §4.2 · p. 6

4.3. Probabilistic time series imputation

• For vector imputation, the method extends conditional diffusion over missing values by using stochastic-process noise in the loss and sampling.The authors posit continuous noise as a more natural inductive bias for irregular time series.Source: §4.3 · p. 7

5.1. Probabilistic modeling

• The experiments used six synthetic datasets, each with 10,000 samples, spanning stochastic processes, dynamical systems, and chaotic systems.The ablation found that modeling interactions across time outperformed treating each time point independently.Source: §5.1 · p. 7
• The transformer-based discriminator could not distinguish the model’s samples from real data, unlike samples from the competing methods.Table 1 reports discriminator accuracy, for which values closer to 0.5 are better.Source: Table 1 · p. 7, §5.1 · p. 8
• Stochastic-process noise achieved lower negative log-likelihood than independent Gaussian noise, especially for complicated samples and smooth curves such as Lorenz.The gap was less visible on noisy CIR data; results were reported for CSPD because it supports likelihood evaluation, with similar ELBO results for DSPD.Source: §5.1 · p. 7
• Across the paper’s datasets, stochastic-process noise did not significantly affect performance relative to independent noise, and sampling speed varied by method.CTFP was fastest, followed by this method and neural ODEs with similar runtime; stochastic-process sampling choices can trade memory impact for parallelization.Source: §5.1 · p. 8

5.2. Forecasting

• The model outperformed TimeGrad on Electricity, Exchange, and Solar while predicting the complete forecast horizon at once.The comparison reports NRMSE and energy score averaged over five runs.Source: Table 2 · p. 8, §5.2 · p. 8

5.3. Neural process

• The authors report achieving 0.737 compared with 0.845, interpreting that comparison as indicating their approach captures the true process, without specifying the measured quantity or evaluation task.They also state that the attentive neural process does not produce the correct uncertainty.Source: §5.3 · p. 9
• Across kernels, the noise process was connected to sample smoothness, while marginal distributions were always correctly captured.Figure 6 illustrates model behavior across different kernels.Source: §5.3 · p. 9

5.4. Imputation

• On Physionet imputation, changing CSDI’s noise source to a Gaussian process improved results, including on this regularly sampled dataset.The authors report statistical significance for the results.Source: §5.4 · p. 9

6. Discussion

• The model can be viewed as a neural stochastic process or as a generative model for solutions to stochastic differential equations.The paper demonstrates its use in forecasting, interpolation, and imputation on both regular and irregularly sampled time series.Source: §6 · p. 9

6.1. Future work

• The authors propose exploring improved diffusion methods, sparse Gaussian processes for larger numbers of points, and learned activations for conditioning models.They also suggest applying the method outside time series, including to point clouds or images, noting its competitiveness on regular grids.Source: §6.1 · p. 9

A.1. Discrete diffusion posterior probability

• The discrete posterior derivation differs from Ho et al. (2020) by using Σ(t) instead of the identity matrix in the covariance.The derivation combines normal transition distributions to obtain a normal posterior.Source: §A.1 · p. 13

A.2. Discrete diffusion loss

• The discrete diffusion loss simplifies to mean squared error between true and predicted noise, as in Ho et al. (2020).The model predicts noise values across the observed time series together, rather than independently for each data point.Source: §A.2 · pp. 13–14

A.3. Continuous diffusion transition probability

• The continuous transition covariance derivation replaces the β(s)I term with β(s)Σ, while the mean derivation remains unchanged.The covariance change follows from using the stochastic process's full covariance matrix.Source: §A.3 · p. 14

A.4. Sampling from an Ornstein-Uhlenbeck process

• The appendix describes modified Wiener, discretized SDE, and multivariate-normal approaches for sampling Ornstein–Uhlenbeck noise.The multivariate-normal approach is selected because the model performs best when predicting standardized noise; its factorization can be reused when the time points do not change.Source: §A.4 · pp. 14–15

A.5. Algorithms

• The DSPD-GP training algorithm draws process noise using a kernel covariance and trains the model to predict standardized noise.The appendix also states that score-based training uses mean squared error on the conditional score, with ODE or SDE sampling.Source: §A.5 · p. 15

B.1.1. DATASETS

• The forecasting experiments use open datasets and six synthetic datasets spanning stochastic processes, dynamical systems, and chaotic systems.The synthetic datasets each contain 10,000 samples; Table 4 describes the open datasets' dimensions, domains, frequencies, training time steps, and prediction lengths.Source: §B.1.1 · p. 15, Table 4 · p. 16

B.1.2. CTFP

• The continuous-time flow process baseline uses a Wiener-process base distribution and invertible transformations to model distributions over functions.Unlike this paper's method, CTFP uses invertible functions and adds noise only at the beginning.Source: §B.1.2 · p. 16

B.1.3. LATENT ODE

• Latent ODE represents a time series as one latent vector and generates its evolution from randomness sampled at the start.The paper contrasts this with its method, which samples random values across the interval and over multiple diffusion steps.Source: §B.1.3 · p. 16

B.1.4. OUR MODELS

• OU/GP noise is mostly better than independent noise on synthetic data by negative log-likelihood.Table 5 reports negative log-likelihood, with lower values indicating better performance.Source: Table 5 · p. 17
• The authors compare feedforward and recurrent models; their reported results indicate that time-dimension interactions matter because models without them learn only marginal distributions and produce poorer samples.Table 6 describes discriminator accuracy, where values near 0.5 indicate samples that are difficult to distinguish from real data.Source: §B.1.4 · pp. 16–17, Table 6 · p. 17

B.2. Multivariate probabilistic forecasting

• Table 7 reports CRPS-sum results for the multivariate probabilistic forecasting task, using non-diffusion baseline values from Salinas et al. (2019b).The supplied passage identifies the metric and baseline source but does not provide comparative values.Source: Table 7 · p. 17

B.3.1. DATASET

• Each GP synthetic series contains 6–49 time points sampled uniformly on [0, 1], with half used as model context and half treated as unobserved.The dataset contains 8,000 training series and 2,000 test series; each series independently samples the RBF kernel σ uniformly from [0.01, 0.05].Source: §B.3.1 · p. 17

B.3.2. MODEL

• The denoising model uses observed points to condition predictions for noisy target points through a learned RBF similarity matrix.It projects observed values pointwise, maps them to target-time dimensions using the similarity matrix, and combines this conditioning representation with projected noisy targets.Source: §B.3.2 · p. 18

• Path probabilities are better when the noise process matches the data’s properties, although marginal distributions are equal across processes and kernel parameters.Source: §B.3.3 · p. 18
• The model reverses the process well, but sampled curves retain process-dependent qualitative properties.Source: §B.3.3 · p. 18
• Increasing OU γ makes curves rougher, whereas increasing GP σ makes them smoother.Source: §B.3.3 · p. 18

• The experiment preserves actual measurement times rather than rounding to the nearest hour, while retaining CSDI and replacing independent normal noise with GP noise.The original setup evaluates the process at regular time points; the comparison still favors the original paper’s one-measurement-per-hour setting.Source: §B.4 · p. 18
• Higher σ values produce better imputation results, but increasing σ does not further improve results.Table 8 reports RMSE averaged over 10 runs and Wilcoxon one-sided-test p-values.Source: §B.4 · p. 18, Table 8 · p. 18
• The 10%-missing case does not produce significant results, which the authors suspect is due to noise.Source: §B.4 · p. 18