PaperlayerView exact revision on arXiv

Source-linked AI summary

Non-autoregressive Conditional Diffusion Models for Time Series Prediction

Lifeng Shen, James Kwok

arXiv:2306.05043v1cs.LG

TL;DR

Time-series diffusion prediction remains difficult because long horizons require modeling complex temporal dependencies, while existing approaches face scalability, error-accumulation, or boundary-disharmony issues. TimeDiff is a non-autoregressive conditional diffusion model that adds future mixup and autoregressive initialization, and experiments across nine real-world datasets show superior overall performance against diffusion models and strong baselines.

  • Problem

    Time-series diffusion models must handle complex, nonlinear, and long temporal dependencies, while efficient denoising and conditioning network design remains open.

  • Method

    TimeDiff uses non-autoregressive conditional diffusion with future mixup and autoregressive initialization tailored to time-series data.

  • Results

    TimeDiff consistently outperforms existing time-series diffusion models and achieves the best overall performance across strong baselines on real-world datasets.

  • Takeaways & Limitations

    TimeDiff provides a time-series diffusion approach that avoids autoregressive decoding while capturing basic patterns and improving long-horizon predictions.

  • Takeaways & Limitations

    TimeDiff has difficulty learning multivariate dependencies when the series contains many variables, such as Traffic.

Abstract

from arXiv · show

Recently, denoising diffusion models have led to significant breakthroughs in the generation of images, audio and text. However, it is still an open question on how to adapt their strong modeling ability to model time series. In this paper, we propose TimeDiff, a non-autoregressive diffusion model that achieves high-quality time series prediction with the introduction of two novel conditioning mechanisms: future mixup and autoregressive initialization. Similar to teacher forcing, future mixup allows parts of the ground-truth future predictions for conditioning, while autoregressive initialization helps better initialize the model with basic time series patterns such as short-term trends. Extensive experiments are performed on nine real-world datasets. Results show that TimeDiff consistently outperforms existing time series diffusion models, and also achieves the best overall performance across a variety of the existing strong baselines (including transformers and FiLM).

1. Introduction

Time series prediction is a challenging conditional-generation problem because real-world series contain complex, nonlinear, and long-range temporal dependencies. TimeDiff addresses this challenge with a non-autoregressive diffusion model using future mixup and autoregressive initialization, outperforming existing diffusion models and strong baselines.

  • Time series prediction is difficult because real-world data involve complex dynamics, nonlinear patterns, and long temporal dependencies, especially over long horizons.
  • Existing diffusion approaches use either autoregressive decoding, which accumulates errors and is slow, or non-autoregressive conditioning in denoising networks’ intermediate layers.
  • TimeDiff introduces future mixup and autoregressive initialization as time-series-specific conditioning mechanisms for non-autoregressive diffusion prediction.Future mixup reveals parts of ground-truth futures during training, while autoregressive initialization supplies basic time-series patterns.
  • Across nine real-world datasets, TimeDiff outperforms existing time-series diffusion models and strong baselines including transformers, DLinear, and FiLM.

2. Preliminaries

Diffusion models generate forecasts by progressively noising and denoising data, with conditional models using past observations to guide future prediction. Existing time-series methods trade off autoregressive error accumulation against non-autoregressive boundary and scalability problems.

  • 2.1. Diffusion Models: Diffusion models transform data into Gaussian noise through a forward process and recover samples through a backward Markovian denoising process.The forward process gradually adds Gaussian noise according to a variance schedule, while the reverse process is modeled by a neural network.
  • 2.2. Conditional DDPMs for Time Series Prediction: Conditional DDPMs forecast future values by applying a denoising model to the forecast window while conditioning on a representation of past observations.The conditioning network F maps the lookback window to c, which guides the reverse diffusion distribution.
  • 2.2. Conditional DDPMs for Time Series Prediction: Designing efficient denoising and conditioning networks for time-series diffusion remains an open question.
  • 2.2. Conditional DDPMs for Time Series Prediction: TimeGrad generates future values autoregressively, causing error accumulation and slow inference that are particularly problematic for long-range prediction.
  • 2.2. Conditional DDPMs for Time Series Prediction: CSDI avoids autoregressive future decoding by diffusing the whole series, but its transformer denoising network has quadratic complexity and its masking can create boundary disharmony.SSSD avoids the quadratic-complexity issue with a structured state space model but retains the non-autoregressive boundary-disharmony problem.

3. Proposed Model

TimeDiff is a conditional non-autoregressive diffusion model whose time-series-specific conditioning combines future mixup with autoregressive initialization. It denoises future sequences from noise while using these conditioning signals to incorporate future information during training and basic temporal patterns during inference.

  • Conditioning the Backward Denoising Process: TimeDiff introduces future mixup and autoregressive initialization as time-series-specific conditioning mechanisms for a non-autoregressive diffusion model.Future mixup combines past information with randomly mixed future information during training, while autoregressive initialization supplies a crude future approximation.
  • Future Mixup: Future mixup randomly mixes past-information embeddings with future ground-truth outputs during training, which are unavailable during inference.The mixing matrix is sampled elementwise from a uniform distribution, and inference omits the future ground-truth segment.
  • Training: During each training step, TimeDiff samples a diffusion step and noise, constructs both conditioning signals, denoises the perturbed future, and updates the denoising network.The training algorithm forms the diffused sample, computes the conditioning signals, predicts the previous diffusion state, and minimizes the corresponding loss.
  • Autoregressive Model: The pretrained linear autoregressive model provides an initial future guess that captures simple patterns such as short-term trends without sequential autoregressive decoding.All forecast columns are obtained simultaneously, avoiding the error accumulation and slow inference associated with autoregressive decoding.
  • Denoising Network: The denoising network combines the diffused input with diffusion-step and conditioning representations through convolutional encoding and decoding layers.The conditioning representation is broadcast over the prediction horizon, concatenated with the input representation, and fused with the conditioning signal before producing the denoised output.
  • Inference: During inference, TimeDiff starts from Gaussian noise and repeatedly applies denoising steps until the final predicted future sequence is obtained.The inference procedure uses the trained denoising and conditioning networks together with the pretrained autoregressive model.

4. Experiments

Experiments compare TimeDiff with diffusion, transformer, and other time-series prediction baselines across nine real-world datasets in univariate and multivariate settings. TimeDiff consistently outperforms existing time-series diffusion models and achieves the best overall performance across the baselines, while better capturing long-term patterns in an ETTh1 example.

  • 4.1. Setup: The evaluation covers nine real-world datasets with hourly, 15-minute, 10-minute, and daily observations spanning energy, traffic, weather, exchange-rate, and wind-power series.The datasets include multivariate series, which are also converted to univariate series for comparison.
  • 4.1. Setup: The benchmark compares TimeDiff with time-series diffusion models, specialized prediction methods, and transformer-based forecasters.Baselines include TimeGrad, CSDI, SSSD, D3VAE, FiLM, Depts, NBeats, PatchTST, Fedformer, Autoformer, and Pyraformer.
  • 4.2. Results: TimeDiff consistently outperforms existing time-series diffusion models and achieves the best overall performance across all baselines in both univariate and multivariate settings.Tables 2 and 3 report testing MSEs for the comparisons.
  • 4.2. Results: CSDI can run out of memory on long multivariate time series because its transformer-based architecture scales poorly with variables and sequence length.The reported memory issue occurs on Traffic and Electricity, and is attributed to CSDI’s transformer use.
  • 4.2. Results: On ETTh1, CSDI and SSSD perform well only in the very short term, whereas TimeDiff better captures the ground-truth trend and periodic patterns over longer horizons.The comparison uses a randomly selected univariate sample and examines steps 192–200 for short-term predictions.

4.3. Ablation Studies

Ablation studies show that both conditioning mechanisms and key design choices improve TimeDiff, while the proposed enhancements also transfer to existing non-autoregressive diffusion models.

  • 4.3.1. Conditioning Mechanism: Both future mixup and autoregressive initialization improve testing MSE, with future mixup especially important on ETTh1.The ETTh1 performance degrades significantly when future mixup is removed.
  • 4.3.2. Different Mixing Strategies: Soft-mixup achieves the best prediction performance without requiring the extra threshold hyper-parameter used by hard-mixup.Hard-mixup is sensitive to the threshold τ.
  • 4.3.3. Predicting xθ vs Predicting ϵθ: Directly predicting data xθ is more effective than predicting diffusion noise ϵθ.The authors suggest nonlinear real-world time-series noise may be confused with diffusion-generated noise.
  • 4.4. Integration into Existing Diffusion Models: Adding future mixup and autoregressive initialization improves CSDI and SSSD, but enhanced models do not consistently match TimeDiff.Enhanced SSSD is comparable to TimeDiff on ETTm1 but remains worse on ETTh1, while CSDI remains below TimeDiff.
  • 4.5. Inference Efficiency: TimeDiff is significantly faster than TimeGrad, CSDI, and SSSD across all tested prediction horizons.TimeGrad is slowest because it uses autoregressive decoding.

5. Conclusion

The paper concludes that TimeDiff combines future mixup and autoregressive initialization to improve time-series diffusion prediction and overall benchmark performance. Its main limitation is difficulty learning multivariate dependencies when the number of variables is large.

  • 5. Conclusion: TimeDiff’s future mixup and autoregressive initialization add useful inductive bias and produce better results than existing time-series diffusion models.Ablation studies support the effectiveness of each component.
  • 5. Conclusion: TimeDiff has difficulty learning multivariate dependencies when time series contain many variables, such as Traffic.The authors propose integrating graph neural networks to address this limitation.

A. Time Series Datasets

The experiments use datasets with differing sampling intervals, temporal dynamics, periodicity, and stationarity properties. Prediction horizons are therefore adjusted by dataset rather than held fixed across all datasets.

  • A. Time Series Datasets: Prediction horizons are adapted to each dataset’s sampling interval, including H = 14 for Exchange and H = 168 for ETTh1.The corresponding horizons represent two weeks for Exchange and one week for ETTh1.
  • A. Time Series Datasets: The datasets exhibit different temporal dynamics, with abundant periodic behavior in Caiso, Traffic, and Electricity.Figure 3 displays only the last variate of each multivariate dataset.
  • A. Time Series Datasets: The ADF test classifies Caiso, ETTm1, ETTh1, and Exchange as non-stationary at a 5% threshold.The reported test uses the null hypothesis that the series is not stationary.

B.1. Network Architecture

TimeDiff uses a conditioning network and an encoder-decoder denoising network built from stacked convolutional blocks. Their input and output sizes are specified by the number of variables and fixed internal dimensions.

  • B.1. Network Architecture: The model comprises conditioning network F plus encoder and decoder subnetworks for denoising.The architecture tables specify each subnetwork’s input and output sizes.
  • B.1. Network Architecture: Each subnetwork is constructed by stacking convolutional blocks with the configuration detailed separately.The internal dimensions d′ and d′′ are both set to 256.

B.2. Baselines

The experiments use publicly available implementations of eight baseline models, including diffusion, transformer, and forecasting methods.

  • Code was downloaded for TimeGrad, CSDI, SSSD, and D3VAE as baseline implementations.
  • The baseline implementations also include FiLM and Depts.
  • NBeats and PatchTST were included among the downloaded baseline implementations.

C. Using Channel-Independence on Multivariate Time Series Datasets

Under channel independence, TimeDiff is compared with PatchTST, FedFormer, Autoformer, and Informer using testing MSEs. TimeDiff still outperforms the other baselines most of the time in this setup.

  • TimeDiff still outperforms PatchTST, FedFormer, Autoformer, and Informer most of the time under channel independence.Each variate of the multivariate time series is predicted independently in this setup.
  • The channel-independence comparison evaluates testing MSEs for each multivariate-series variate predicted independently.
  • PatchTST did not use channel independence in the Section 4.1 experiments, but it is compared under that setup in this section.
Loading 2306.05043v1…