Source-linked AI summary
Autoregressive Denoising Diffusion Models for Multivariate Probabilistic Time Series Forecasting
Kashif Rasul, Calvin Seward, Ingmar Schuster, Roland Vollgraf
TL;DR
Multivariate probabilistic forecasting needs models that represent uncertainty and dependencies across many correlated series. TimeGrad combines autoregressive forecasting with diffusion-based energy modeling, learning denoising gradients through a variational likelihood bound and sampling future distributions from white noise. Across six benchmarks, the authors report new state-of-the-art performance against competitive methods.
Problem
Multivariate forecasts must model uncertainty and dependencies across series, while full covariance modeling becomes impractical as dimensionality grows.
Method
TimeGrad autoregressively learns and samples each next-step multivariate distribution with an energy-based diffusion model trained using a variational likelihood bound.
Results
TimeGrad establishes the new state-of-the-art against competitive methods on six commonly used time series benchmarks.
Takeaways & Limitations
The autoregressive energy-based combination retains future extrapolation capability, flexible high-dimensional distribution modeling, and computational tractability.
Takeaways & Limitations
The paper identifies explicit modeling of discrete distributions as future work, since the EBM setting does not require dequantization but does not yet provide that modeling.
Abstract
from arXiv · showhide
In this work, we propose \texttt{TimeGrad}, an autoregressive model for multivariate probabilistic time series forecasting which samples from the data distribution at each time step by estimating its gradient. To this end, we use diffusion probabilistic models, a class of latent variable models closely connected to score matching and energy-based methods. Our model learns gradients by optimizing a variational bound on the data likelihood and at inference time converts white noise into a sample of the distribution of interest through a Markov chain using Langevin sampling. We demonstrate experimentally that the proposed autoregressive denoising diffusion model is the new state-of-the-art multivariate probabilistic forecasting method on real-world data sets with thousands of correlated dimensions. We hope that this method is a useful tool for practitioners and lays the foundation for future research in this area.
1. Introduction
Multivariate probabilistic forecasting must capture uncertainty and dependencies across series, motivating flexible global models such as TimeGrad. TimeGrad combines autoregressive forecasting with energy-based modeling to remain expressive and tractable.
- Classical methods often produce univariate point forecasts, use hand-tuned seasonal features, and train each series separately.
- Deep learning alternatives provide global end-to-end training, automatic feature extraction, and easier incorporation of exogenous covariates.
- Uncertainty modeling matters for downstream business decision making, while individual series are often statistically dependent.
- Flexible multivariate distributions require probabilistic methods with tractable likelihoods rather than restricting the data to simple distribution classes or low-rank approximations.
- TimeGrad uses autoregressive energy-based models to combine autoregressive extrapolation with flexible high-dimensional distribution modeling while remaining computationally tractable.
2. Diffusion Probabilistic Model
Diffusion probabilistic models learn a reverse denoising process for data corrupted by a fixed Gaussian-noise Markov chain. Training uses a variational likelihood bound, while sampling progressively transforms white noise into data samples through Langevin-like updates.
- The model represents pθ(x0) as a latent-variable distribution whose posterior is fixed to a forward Markov chain that gradually adds Gaussian noise.
- The forward process uses an increasing variance schedule β1, ..., βN, while the learned reverse process starts from N(xN; 0, I) and removes noise through Gaussian transitions.
- The reverse-transition parameters take the noisy vector xn and noise index n as inputs, and training fits the data distribution by minimizing a variational-bound objective.
- The forward process permits closed-form sampling at arbitrary noise levels using αn := 1 − βn and cumulative product ᾱn.
- Instead of predicting the posterior mean directly, the network predicts Gaussian noise ε from xn, simplifying the objective to a score-matching-like loss.
- At inference, sampling begins with white noise xN and iteratively reduces the noise scale through reverse-process updates resembling Langevin dynamics.
3. TimeGrad Method
TimeGrad models each future multivariate time step conditionally on past observations, covariates, and an RNN state, using a denoising diffusion model within an autoregressive forecast.
- A factorizing output distribution misses dependencies among entities, while a full Gaussian covariance costs O(D^2) parameters and O(D^3) loss computation.Low-rank covariance approximations reduce this burden but restrict the modeled dependencies to second-order effects.
- TimeGrad targets the conditional distribution of future multivariate time steps given past observations and covariates.
- Each conditional factor is learned with a conditional denoising diffusion model whose denoising network is conditioned on the RNN hidden state.Training randomly samples context and prediction windows and minimizes the model’s negative log-likelihood.
- The model uses an LSTM or GRU RNN to encode the sequence up to time t, incorporating covariates through the hidden state.
- At inference, TimeGrad samples the next step from Gaussian noise through annealed Langevin dynamics, feeds it autoregressively into the RNN, and repeats across the forecast horizon.Repeated trajectory sampling, such as S = 100, produces empirical uncertainty quantiles.
- Entity-wise rescaling divides inputs by their context-window mean and reverses that transformation on generated samples.The paper reports that this normalization simplifies the problem and improves empirical performance.
4. Experiments
TimeGrad is evaluated on six real-world multivariate data sets using CRPS and CRPSsum, with comparisons against classical and deep-learning baselines. Experiments also examine diffusion length and illustrate probabilistic forecasts on high-dimensional Traffic data.
- Evaluation Metric: The evaluation uses CRPS on each dimension and CRPSsum after summing all dimensions, with lower CRPSsum indicating better performance.CRPS is a proper scoring function, and CRPSsum is used because some baselines lack analytical forecast distributions or meaningful likelihoods.
- Data Sets: Experiments use six open data sets—Exchange, Solar, Electricity, Traffic, Taxi, and Wikipedia—with dimensions, frequencies, training lengths, and prediction horizons summarized in Table 1.Traffic does not require scale normalization, unlike the other data sets.
- Baselines: TimeGrad is compared with classical multivariate models, including VAR, VAR-Lasso, GARCH, VES, and low-rank or copula-based methods.The comparison also includes GP-scaling, GP-Copula, and Vec-LSTM-lowrank-Copula.
- Baselines: The comparison also includes deep-learning baselines such as KVAE, Vec-LSTM-ind-scaling, and Transformer-MAF.Transformer-MAF combines Transformer temporal conditioning with a masked autoregressive flow emission model.
- Results: TimeGrad achieves the new state-of-the-art on all but the smallest benchmark data set according to Table 2 CRPSsum values.The reported TimeGrad means and standard errors are based on 10 independent retraining and evaluation runs.
- Ablation: Electricity ablations show good performance by N ≈10, an optimum at N ≈100, and no benefit from larger diffusion lengths when other hyperparameters remain fixed.The study varies N from 2 through 256 while holding other hyperparameters unchanged.
5. Related Work
Related work spans score-based energy models, diffusion methods for waveform generation, and neural probabilistic time-series forecasting. TimeGrad differs by modeling temporal development autoregressively while addressing heterogeneous multivariate entities.
- Energy-Based Methods: Energy-based diffusion models learn gradients of log-density and use Langevin dynamics to sample complicated distributions without adversarial optimization.These methods have achieved strong results in image generation.
- Diffusion Methods: Recent conditional denoising-diffusion methods generate high-fidelity vector-valued waveforms but do not model temporal development as TimeGrad does.Waveform dimensions are typically highly correlated and uniformly scaled, unlike arbitrary multivariate time-series entities.
- Neural Time-Series Forecasting: Multivariate neural forecasting methods include point forecasts and probabilistic models based on Gaussian copulas, GANs, and related distributional approaches.The broader literature also includes univariate neural forecasting methods.
6. Conclusion and Future Work
The paper presents TimeGrad as an autoregressive EBM-based method for sampling multivariate next-step distributions and reports state-of-the-art results across six benchmarks. It identifies sampling efficiency, discrete data, anomaly detection, and longer-sequence architectures as directions for future work.
- Conclusion: TimeGrad learns and autoregressively samples the distribution of each multivariate time-series next step using energy-based models.The conclusion describes the method as versatile for multivariate probabilistic forecasting.
- Conclusion: Experiments on six commonly used benchmarks establish new state-of-the-art performance against competitive methods.This is the paper’s summarized empirical conclusion.
- Future Work: Sampling loops over the EBM approximator N times, while improved variance schedules and L1 loss may reduce sampling steps at a small quality cost.Training does not require looping over the EBM approximator as in normalizing-flow stacks.
- Future Work: Future work could explicitly model discrete distributions, evaluate TimeGrad for anomaly detection, and replace the RNN with Transformers for long sequences.Graph neural networks are also suggested when entity relationships provide a useful inductive bias.