PaperlayerView exact revision on arXiv

Source-linked AI summary

Predict, Refine, Synthesize: Self-Guiding Diffusion Models for Probabilistic Time Series Forecasting

Marcel Kollovieh, Abdul Fatir Ansari, Michael Bohlke-Schneider, Jasper Zschiegner, Hao Wang, Yuyang Wang

arXiv:2307.11494v3cs.LGcs.AIstat.ML

TL;DR

Time-series diffusion models have often been conditional and tailored to particular forecasting or imputation tasks. TSDiff is trained unconditionally and uses inference-time self-guidance, applying the model to forecasting, forecast refinement, and synthetic-data generation. It is competitive with task-specific forecasting methods, and its synthetic samples support strong downstream forecasting results.

  • Problem

    Prior time-series diffusion models were task-specific conditional models for forecasting or imputation, rather than unconditional models usable across downstream tasks.

    Source: §1 · p. 1

  • Method

    TSDiff is an unconditional time-series diffusion model conditioned at inference through self-guidance, and its learned implicit density is also used to refine base-forecaster predictions.

    Source: §1 · p. 2, §3 · p. 4, §3.2 · p. 5

  • Results

    TSDiff is competitive with task-specific forecasting baselines; synthetic samples outperform TimeVAE and TimeGAN by LPS and yield stronger DeepAR and Transformer results on most datasets.

    Source: §1 · p. 2, §4.3 · p. 9, §6 · p. 10

  • Takeaways & Limitations

    The reported results span inference-time forecasting and refinement as well as training downstream forecasters on synthetic data from TSDiff.

    Source: §1 · p. 2, §6 · p. 10

    Observation self-guidance has high computational cost because it uses an iterative denoising process.

    Source: §6 · p. 10

Abstract

from arXiv · show

Diffusion models have achieved state-of-the-art performance in generative modeling tasks across various domains. Prior works on time series diffusion models have primarily focused on developing conditional models tailored to specific forecasting or imputation tasks. In this work, we explore the potential of task-agnostic, unconditional diffusion models for several time series applications. We propose TSDiff, an unconditionally-trained diffusion model for time series. Our proposed self-guidance mechanism enables conditioning TSDiff for downstream tasks during inference, without requiring auxiliary networks or altering the training procedure. We demonstrate the effectiveness of our method on three different time series tasks: forecasting, refinement, and synthetic data generation. First, we show that TSDiff is competitive with several task-specific conditional forecasting methods (predict). Second, we leverage the learned implicit probability density of TSDiff to iteratively refine the predictions of base forecasters with reduced computational overhead over reverse diffusion (refine). Notably, the generative performance of the model remains intact -- downstream forecasters trained on synthetic samples from TSDiff outperform forecasters that are trained on samples from other state-of-the-art generative time series models, occasionally even outperforming models trained on real data (synthesize).

1 Introduction

Prior time-series diffusion models were task-specific conditional models, limiting their applicability to the tasks they were trained for and foregoing unconditional generation. TSDiff uses an unconditional model with inference-time self-guidance for forecasting, refinement, and synthetic-data generation.

1 Introduction

  • Time-series forecasting informs decisions in finance, renewable energy, and healthcare.Deep learning approaches frame forecasting as generating the future conditioned on the past.Source: §1 · p. 1
  • Prior time-series diffusion models were task-specific, limiting their applicability to trained tasks and foregoing unconditional generative capabilities [44] [52] [1] [6].The paper asks whether one unconditional diffusion model can address multiple downstream tasks, including conditional ones.Source: §1 · p. 1
  • TSDiff is trained unconditionally and uses self-guidance at inference to condition forecasts without auxiliary networks or changes to training.The approach is presented for arbitrary forecasting and imputation tasks.Source: §1 · p. 2
  • The self-guidance approach is competitive with task-specific conditional models across multiple datasets and forecasting scenarios, without conditional training.The paper also proposes iterative refinement of base-forecaster predictions using TSDiff’s implicit density, with reduced computational overhead relative to reverse diffusion.Source: §1 · p. 2
  • Forecasters trained on TSDiff samples outperform those trained on TimeVAE and TimeGAN samples by LPS, and sometimes outperform forecasters trained on real samples.The paper defines Linear Predictive Score (LPS) as the test forecast performance of a linear ridge regression model trained on synthetic samples.Source: §1 · p. 2

2 Background

The background describes diffusion as a latent-variable generative process with forward and learned reverse steps, then introduces classifier guidance for conditional generation.

2.1 Denoising Diffusion Probabilistic Models

  • A DDPM represents data generation with latent variables generated through a fixed Gaussian Markov forward process.The forward process permits direct sampling of noisy states from an observed datapoint.Source: §2.1 · p. 3
  • The reverse process approximates the true reverse transitions by maximizing an ELBO approximation, using a denoising network trained to predict sampled noise.The objective can be used to train unconditional or conditional models by incorporating conditioning input into the denoising network.Source: §2.1 · p. 3

2.2 Diffusion Guidance

  • Classifier guidance uses Bayes-rule score decomposition and an auxiliary classifier to estimate the class-conditioning gradient for conditional sampling.A scale parameter controls guidance strength.Source: §2.2 · p. 3

3 TSDiff: an Unconditional Diffusion Model for Time Series

TSDiff trains an unconditional model over complete time series, then conditions it at inference on observed and target timesteps. The section develops observation self-guidance and iterative refinement using the model’s learned density.

3 TSDiff: an Unconditional Diffusion Model for Time Series

  • TSDiff models the conditional distribution of target timesteps given observed values, including when the observed subsequence is noncontiguous.A single unconditional model is trained and conditioned at inference for these distributions.Source: §3 · p. 4

3.1 Observation Self-Guidance

  • Observation self-guidance reuses the denoising model to estimate the series and guide reverse diffusion, requiring no auxiliary network or training changes.The guidance distribution is parameterized by the estimated series and the observations’ log-likelihood.Source: §3.1 · pp. 4–5, Figure 2 · p. 4
  • Quantile self-guidance uses an asymmetric Laplace loss and guides sampling at evenly spaced κ levels to better represent forecast quantiles.This addresses the limitation of MSE, which quantifies average quadratic deviation from the mean rather than all quantiles.Source: §3.1 · p. 5

3.2 Prediction Refinement

  • Prediction refinement starts from any base forecaster’s sample and uses Langevin sampling in data space to balance TSDiff likelihood with closeness to that forecast.The initial series combines observed values and the base forecast; the regularizer can use MSE or quantile loss.Source: §3.2 · pp. 5–6
  • The alternative likelihood-maximization formulation uses gradient descent, but its non-convex objective gives no global-optimum guarantee and makes the result depend on initialization.The paper reports observing this initialization dependence in its experiments.Source: §3.2 · p. 6
  • A single representative diffusion step approximates the log-likelihood instead of sampling several steps, improving refinement stability and computed once per dataset.The representative step is selected using the diffusion-model training loss.Source: §3.2 · p. 6

4 Experiments

Experiments assess whether unconditional time-series diffusion can support forecasting, forecast refinement, and synthetic-data training. Across these tasks, TSDiff is competitive with conditional models and improves forecasts or downstream performance in the reported comparisons.

4 Experiments

  • The experiments evaluate probabilistic forecasts on eight univariate datasets using CRPS and compare observation self-guidance with statistical, deep-learning, and conditional-diffusion baselines.The authors state that the goal was to demonstrate the efficacy of unconditional models against task-specific conditional models, not to obtain state-of-the-art forecasting results on the datasets.Source: §4 · p. 6, §4.1 · p. 7

4.1 Forecasting using Observation Self-Guidance

  • TSDiff-Q achieves the lowest or second-lowest CRPS on 5/8 datasets while remaining competitive with task-specific conditional models.Table 1 reports forecasting results across eight benchmark datasets.Source: §4.1 · p. 7, Table 1 · p. 7
  • TSDiff-Q remains competitive with task-specific conditional models when historical context has missing values during inference.The missing-value results cover six benchmark datasets; the tested cases include random missingness and beginning- or end-of-context blackout.Source: §4.1 · pp. 7–8, Table 2 · p. 8

4.2 Refining Predictions of Base Forecasters

  • At least one refinement method lowers base-forecast CRPS on every dataset; energy-based refinement generally favors point forecasters, while ML-Q favors probabilistic forecasters.Experiments refined Seasonal Naive, Linear, DeepAR, and Transformer forecasts for 20 steps; refinement has lower computational overhead than self-guidance but depends strongly on the base forecaster.Source: §4.2 · pp. 8–9, Table 3 · p. 8

4.3 Training Downstream Models using Synthetic Data

  • TSDiff samples significantly outperform TimeGAN and TimeVAE on LPS; DeepAR and Transformer trained on them outperform baselines on most datasets and score reasonably against real-data-trained forecasters.Real-data-trained forecasters had access to time features and lags extending beyond TSDiff’s modeled sequence length.Source: §4.3 · p. 9, Table 4 · p. 9

5 Related Work

Related work places TSDiff alongside diffusion applications beyond images, time-series diffusion models, and methods for diffusion guidance. It distinguishes TSDiff by training unconditionally and applying guidance at inference without auxiliary networks or training changes.

Diffusion Models

  • Diffusion models have been applied to audio, protein, and graph generation, and to image inpainting; TSDiff’s architecture modifies DiffWave, originally developed for audio synthesis.Source: §5 · p. 10

Diffusion Models for Time Series

  • Prior time-series diffusion models such as TimeGrad, CDSI, and SSSD are conditional and trained for particular forecasting or imputation tasks, unlike TSDiff’s unconditional training and inference-time guidance.Source: §5 · p. 10

Diffusion Guidance

  • Unlike classifier guidance or jointly trained conditional-unconditional guidance, observation self-guidance needs no auxiliary network or training-procedure modification.The paper also applies diffusion guidance to time series, whereas prior works discussed here primarily focus on images.Source: §5 · p. 10

6 Conclusion

TSDiff uses unconditional training and inference-time self-guidance for probabilistic forecasting, while its learned density also supports forecast refinement and synthetic data generation. The conclusion highlights competitive forecasting and synthetic-sample predictive quality, alongside the computational cost of iterative denoising.

6 Conclusion

  • TSDiff with self-guidance is competitive with strong task-specific forecasting baselines, and its synthetic samples yield stronger downstream predictive quality than alternative generative models.The authors also report that TSDiff-trained forecasters occasionally outperform those trained on real data.Source: §6 · p. 10
  • Iterative denoising’s high computational cost is the key limitation of observation self-guidance.The authors evaluated TSDiff for probabilistic forecasting; they describe imputation and other inverse problems as potential applications.Source: §6 · p. 10

A.1 Asymmetric Laplace Distribution

  • The asymmetric Laplace distribution uses quantile level κ, location m, and scale b; κ = 0.5 yields the symmetric Laplace distribution.Its negative log-likelihood corresponds to pinball loss.Source: §A.1 · p. 15

A.2 Algorithms

  • Observation self-guidance starts from Gaussian noise and iteratively denoises it toward the observed values using guided reverse diffusion.The procedure takes an observation and guidance scale as inputs.Source: §A.2 · p. 15
  • Energy-based refinement combines observations with a base forecaster’s prediction, then iteratively updates the initial guess using the conditional distribution’s score.Setting γ = 0 gives the maximum-likelihood refinement scheme.Source: §A.2 · p. 15

B.1 Datasets

The experiments use eight popular univariate time-series datasets from different domains, with daily or hourly frequencies. Sequences are formed by random-time slicing at dataset-specific lengths.

B.1 Datasets

  • The benchmark covers eight univariate datasets; sequences have length 360 for hourly data and 390 for daily data, sampled at random timesteps.The dataset overview is provided in Table 5.Source: §B.1 · p. 16, Table 5 · p. 16
  • Solar, Electricity, and Traffic cover Alabama solar production, 370 customers’ electricity use, and San Francisco freeway occupancy.The Solar data are from 2006; the Traffic data cover 2015–2016.Source: §B.1 · p. 16
  • The remaining datasets are Exchange, M4, KDDCup [16], UberTLC, and Wikipedia, covering exchange rates, hourly competition data, air quality, taxi pickups, and page hits.Exchange spans daily rates for eight countries from 1990 to 2016; UberTLC covers NYC pickups from January to June 2015.Source: §B.1 · p. 16

B.2 Metrics

The section describes CRPS for probabilistic forecast evaluation and introduces LPS to assess the predictive quality of synthetic time-series samples.

B.2 Metrics

  • CRPS evaluates probabilistic forecasts by measuring compatibility between the predicted quantile function and the observation.It is a proper scoring rule and is defined as the integral of pinball loss from 0 to 1.Source: §B.2 · p. 17
  • In practice, CRPS is approximated from samples using nine quantile levels from 0.1 to 0.9 and 100 sample forecasts.The implementation uses GluonTS defaults.Source: §B.2 · p. 17
  • LPS is the test CRPS of a ridge regression model trained on 10,000 synthetic samples, measuring their predictive quality.For a point forecaster such as the linear model, CRPS equals the 0.5-quantile loss, also called Normalized Deviation.Source: §B.2 · p. 17

B.3 Hyperparameters and Training Details

The appendix reports TSDiff training and inference settings, including normalization, guidance-scale selection, missing-value evaluation, and refinement choices. It also describes the representative diffusion-step approximation and the source of self-guidance’s inference overhead.

B.3 Hyperparameters and Training Details

  • TSDiff used 1,000 Adam epochs, learning rate 1e-3, 128 batches of 64 sequences per epoch, three residual layers with 64 channels, and 100 diffusion steps.Some datasets used an input-to-output skip connection; diffusion timesteps used 128-dimensional positional embeddings, with a linear schedule from β1 = 0.0001 to β100 = 0.1.Source: §B.3 · p. 17
  • Mean guidance used scale 4/32 on every dataset; quantile guidance scales were tuned on validation data, and TSDiff-Q samples aligned closely with observations.The tested mean-guidance scales were 2/32, 3/32, 4/32, and 5/32; the quantile-guidance choices were 1, 2, 4, and 8.Source: §B.3 · p. 18, Figure 5 · p. 18, Table 7 · p. 18
  • Missing-value evaluation masked 50% of context timesteps and used self-guidance to predict both missing values and the forecast; M4 and Wikipedia were excluded because their remaining series were too short.CRPS was computed only for the forecast horizon, and a single unconditional TSDiff-Q model was used per dataset.Source: §B.3 · p. 18, Figure 6 · p. 19
  • Observation self-guidance’s extra inference overhead comes from computing a neural-network gradient; inference times were compared with TSDiff-Cond on Exchange.The comparison controlled for batch size, sample count, and GPU.Source: §B.3 · p. 19, Table 8 · p. 19
  • A representative diffusion step approximates the ELBO using the step that best approximates average loss; its loss profile was computed from 1,024 datapoints in about 13 seconds per dataset.Refinement showed immediate improvement after one iteration for Seasonal Naive and Linear, while DeepAR and Transformer improved gradually; 20 iterations were selected for all datasets and base forecasters.Source: §B.3 · p. 19

B.4 Baselines

The forecasting experiments compare TSDiff with statistical and deep-learning baselines, alongside conditional diffusion alternatives and generative models used in synthetic-data experiments. The section also describes TSDiff-Cond and supplementary loss and refinement plots.

B.4 Baselines

  • The forecasting experiments compare against eleven statistical and deep-learning baselines.Supplementary figures plot loss by diffusion step across eight datasets and relative CRPS over refinement iterations on Solar.Source: §B.4 · p. 19, Figure 7 · p. 20, Figure 8 · p. 20
  • Seasonal Naive, ARIMA, ETS, and Linear provide simple forecasting comparisons, ranging from last-season values to ridge regression across time.Linear uses 10,000 randomly sampled sequences per dataset.Source: §B.4 · pp. 20–21
  • Deep-learning forecasting baselines include TFT, DeepAR, MQ-CNN, DeepState, and Transformer models.Their described approaches include attention, autoregression, convolution, and recurrent networks paired with linear dynamical systems.Source: §B.4 · p. 21
  • CSDI is a conditional diffusion baseline, while TSDiff-Cond injects observed context and a mask through Conv1x1 layers.Compared with SSSD, TSDiff-Cond uses S4 layers before conditioning addition, treats lagged values as univariate features, and computes loss over the entire series; this led to better conditional-model performance in these experiments.Source: §B.4 · pp. 21–22
  • TimeVAE and TimeGAN are the two generative-model comparators in train-on-synthetic, test-on-real experiments.TimeVAE uses variational autoencoders with trend and seasonality components; TimeGAN trains a generative adversarial network in a latent space.Source: §B.4 · p. 22

C Synthetic Samples

TSDiff’s synthetic time series closely resemble real samples across the evaluated datasets and are described as higher quality than TimeVAE and TimeGAN samples. Examples include realistic solar patterns and greater diversity than some baseline samples.

C Synthetic Samples

  • TSDiff samples closely resemble real time series and are higher quality than TimeVAE and TimeGAN samples across the datasets shown.The figure set compares real and generated samples for Solar, Electricity, Traffic, Exchange, M4, UberTLC, KDDCup, and Wikipedia.Source: C Synthetic Samples · p. 22, Figure 10 · p. 22, Figure 11 · p. 23
  • On Solar, TSDiff captures nighttime zero production and varied peaks, while baseline samples are more homogeneous and miss zero values precisely.TimeGAN also shows mode collapse on Electricity, M4, and UberTLC, whereas TSDiff samples remain diverse and realistic.Source: C Synthetic Samples · p. 22, Figure 10 · p. 22, Figure 11 · p. 23
Loading 2307.11494v3…