PaperlayerView exact revision on arXiv

Source-linked AI summary

Multivariate Probabilistic Time Series Forecasting via Conditioned Normalizing Flows

Kashif Rasul, Abdul-Saboor Sheikh, Ingmar Schuster, Urs Bergmann, Roland Vollgraf

arXiv:2002.06103v3cs.LGstat.ML

TL;DR

High-dimensional probabilistic forecasting must represent uncertainty and dependencies among interacting time series without relying on costly or overly simple distributions. The paper combines autoregressive deep learning with conditioned normalizing flows, achieving state-of-the-art results across diverse real-world benchmarks while leaving discrete ordinal flow modeling as an open problem.

  • Problem

    Probabilistic multivariate forecasting must capture uncertainty and cross-series dependencies, while conventional full-covariance models become difficult and expensive as dimensionality grows.

  • Method

    The paper combines an autoregressive model, such as an RNN or attention module, with a conditioned normalizing flow for high-dimensional multivariate temporal dynamics.

  • Results

    MAF with RNN or self-attention achieves state-of-the-art CRPSsum on all benchmarks, while bipartite-flow RNN models are competitive or better than prior results.

  • Takeaways & Limitations

    The combination retains autoregressive extrapolation ability while providing flexible density estimation for broad high-dimensional data distributions and computational efficiency.

  • Takeaways & Limitations

    Modeling discrete ordinal data via flows remains an open problem, limiting direct coverage of data such as sales.

Abstract

from arXiv · show

Time series forecasting is often fundamental to scientific and engineering problems and enables decision making. With ever increasing data set sizes, a trivial solution to scale up predictions is to assume independence between interacting time series. However, modeling statistical dependencies can improve accuracy and enable analysis of interaction effects. Deep learning methods are well suited for this problem, but multivariate models often assume a simple parametric distribution and do not scale to high dimensions. In this work we model the multivariate temporal dynamics of time series via an autoregressive deep learning model, where the data distribution is represented by a conditioned normalizing flow. This combination retains the power of autoregressive models, such as good performance in extrapolation into the future, with the flexibility of flows as a general purpose high-dimensional distribution model, while remaining computationally tractable. We show that it improves over the state-of-the-art for standard metrics on many real-world data sets with several thousand interacting time-series.

1 INTRODUCTION

The paper addresses probabilistic forecasting for interacting multivariate time series, where uncertainty and cross-series dependence matter for trustworthy decisions and accurate predictions. It proposes scalable autoregressive deep learning models with conditioned normalizing flows.

  • Probabilistic forecasts distinguish low-noise from high-noise regions, which matters when predictions support anomaly detection or business decisions.Point estimates ignore risk arising from forecast noise.
  • Multivariate models must capture dependencies such as product cannibalization and disruptions propagating across nearby traffic sensors.
  • The proposed architectures combine autoregressive deep learning with normalizing flows to model multivariate temporal dynamics end to end.
  • The models scale to thousands of interacting time series and adapt to broad underlying distributions through normalizing flows.
  • Transformer-based models are computationally efficient during training because attention layers operate in parallel.
  • The paper evaluates dependency learning on toy data and forecasting performance against competitive baselines across diverse real-world data sets.

2 BACKGROUND

Normalizing flows transform complex data distributions into simple latent distributions through invertible mappings with tractable likelihoods. The background also covers conditional flows, coupling layers, training, dequantization, and Transformer attention for sequence modeling.

  • Density estimation via normalizing flows: Normalizing flows map data in R^D to a simple distribution in R^D using a sequence of bijections.The inverse is easy to evaluate, and the Jacobian determinant can be computed in O(D) time.
  • Density estimation via normalizing flows: Real NVP coupling layers leave part of the input unchanged and transform the remainder using scaling and translation functions.Alternating transformed and unchanged dimensions across layers produces a nonlinear density map.
  • Training: Flow likelihoods are computed with the change-of-variables formula and optimized by maximizing average log likelihood with stochastic gradient descent.
  • Density estimation via normalizing flows: MAF makes the transformation autoregressive, yielding a triangular Jacobian whose determinant is tractable.
  • Conditional flows: Conditioning information h enters the scaling and translation networks, enabling conditional density modeling pX(x|h).
  • Discrete data: Uniform[0, 1) dequantization avoids unbounded likelihoods when normalizing flows model discrete data.The resulting continuous model’s log-likelihood lower-bounds the discrete model’s log-likelihood.
  • Transformer sequence modeling: Transformers use masked multi-head self-attention to capture temporal dependencies while preventing attention to future information.Encoder-decoder configurations can autoregressively generate outputs from warm-up sequences.

3 RELATED WORK

Related work combines sequential models with flow-based or other probabilistic density models, while high-dimensional multivariate forecasting often uses approximations to remain tractable. The paper positions its approach against these alternatives.

  • Transformation Autoregressive Networks model multivariate densities as products of conditional distributions whose parameters derive from an RNN state.
  • PixelSNAIL models joint distributions with causal convolutions and self-attention to capture long-term temporal dependencies.
  • Bipartite flows with RNN temporal conditioning provide a related conditional generative model for multivariate sequential data.The related approach can also condition on observed covariates represented in the latent space.
  • Gaussian Copula Processes combine RNNs, low-rank covariance, and non-Gaussian marginals to scale to thousands of series with state-of-the-art results.

4 TEMPORAL CONDITIONED NORMALIZING FLOWS

The paper models the conditional joint distribution of interacting multivariate time series with autoregressive temporal conditioning and a flexible normalizing flow. It combines RNN- or Transformer-based conditioning with scalable training and iterative flow-based sampling.

  • A multivariate time series is represented as xt ∈ R^D over a context window [1, t0) and prediction window [t0, T].
  • The model addresses the limitation of factorized emissions, which cannot capture dependencies between series, while full covariance modeling scales poorly with O(D^2) parameters.
  • The conditional joint distribution pX(xt|ht; θ) is modeled with a Real NVP or MAF flow conditioned on an RNN hidden state or an attention-based embedding of previous observations.
  • 4.1 TRAINING: The Transformer uses an encoder-decoder architecture with causal masking, positional encodings as covariates, and parallel training across x1:T.
  • 4.1 TRAINING: The Transformer has complexity O(T^2D) versus O(TD^2) for the RNN and is computationally smaller when D > T.
  • At inference, the flow maps Gaussian noise backward into the next observation, which is fed recursively into the RNN or decoder for subsequent conditioning states.

5 EXPERIMENTS

Experiments evaluate the model first on a simulated pipe-flow system and then on six real-world data sets. The model captures spatial and temporal dependencies in the toy system, while MAF-based variants achieve state-of-the-art CRPSsum across all real-world benchmarks and Transformer-MAF reflects observed dependency structure during extrapolation.

  • Experimental setup: The experiments combine a toy pipe-flow sanity check with probabilistic forecasting evaluations on six real-world data sets.The real-world data sets are Exchange, Solar, Electricity2, Traffic, Taxi, and Wikipedia.
  • Simulated flow in a system of pipes: The simulated system tests whether the model learns relations between time series by modeling liquid flow through sensors and valves.The setup uses Gamma-distributed initial pressure and independently sampled Beta-distributed valves.
  • Simulated flow in a system of pipes: The model captures the positive covariance between S1 and S2 caused by their shared source S0.This covariance is evaluated at a fixed time point and shown in the left covariance matrix of Figure 4.
  • Simulated flow in a system of pipes: The learned cross-covariance reproduces liquid flow from S0 at time t to S1 and S2 at t + 1, then to S3 at t + 2.The comparison uses cross-covariance matrices between consecutive time points in the ground truth and trained model.
  • Real-world data sets: MAF with either RNN or self-attention temporal conditioning achieves state-of-the-art CRPSsum on all six benchmarks.Bipartite flows with RNN are also reported as outperforming or competing with previous state-of-the-art results; CRPS and MSE are evaluated additionally.
  • Real-world data sets: Transformer-MAF extrapolation reflects most of the observed Traffic cross-covariance structure, especially among highly correlated sensors.The comparison uses the mean of 100 sampled trajectories and shows mostly small absolute differences from the ground-truth matrix.

6 CONCLUSION

The paper combines conditional normalizing flows with autoregressive models to represent high-dimensional probabilistic multivariate time series. It identifies more expressive flow designs and discrete ordinal data as important directions for future work.

  • 6 CONCLUSION: Conditional normalizing flows combined with autoregressive models provide a general method for high-dimensional probabilistic multivariate time-series modeling.The autoregressive component can use a recurrent neural network or attention module, while the flow avoids a fixed simple distribution class.
  • 6 CONCLUSION: Autoregressive models support forecasting and extrapolation into the future, while flow models provide flexible distribution modeling.
  • 6 CONCLUSION: MAF improved performance over Real NVP, motivating better flow architectures, dequantization methods, coupling layers, and conditioning mechanisms.The passage attributes the improvement to Real NVP's weaker density-modeling performance compared with MAF.
  • 6 CONCLUSION: Flow++-style expressive bipartite models and methods for discrete categorical data remain future work to assess.
  • 6 CONCLUSION: Modeling discrete ordinal data via flows remains an open problem.

A DATA SET DETAILS

The appendix describes the properties of the data sets used in the experiments.

  • A DATA SET DETAILS: The experiments use data sets whose properties are summarized in Table 2.
  • A DATA SET DETAILS: Table 2 serves as the appendix reference for the experimental data-set properties.
  • A DATA SET DETAILS: The data-set descriptions are organized as a table of properties used in the experiments.

B ADDITIONAL METRICS

The appendix reports additional probabilistic forecasting metrics and comparisons against classical methods and prior models. Results use CRPS-based evaluations, with some model estimates averaged over repeated runs.

  • B ADDITIONAL METRICS: Metrics were evaluated using the same open-source code provided by Salinas et al. (2019a).
  • B ADDITIONAL METRICS: Table 3 reports test-set CRPSsum, comparing classical baselines with the Transformer-MAF model.The baselines include VAR, VAR-Lasso, GARCH, GP, KVAE, and VES.
  • B ADDITIONAL METRICS: Table 4 compares test-set marginal CRPS for prior models and the LSTM-Real-NVP, LSTM-MAF, and Transformer-MAF models.The metric averages marginal CRPS over dimensions and predicted time steps relative to the test interval.

B.3 MEAN SQUARED ERROR (MSE)

The appendix defines marginal MSE over all series dimensions and the full prediction range, then compares the proposed models with univariate and multivariate forecasting methods. The reported results improve on the compared metrics for Traffic and show the same pattern on other data sets.

  • B.3 MEAN SQUARED ERROR (MSE): MSE is the mean squared error across all time-series dimensions D and the entire prediction range relative to test data.
  • B.3 MEAN SQUARED ERROR (MSE): Table 5 compares test-set MSE for prior models with LSTM-Real-NVP, LSTM-MAF, and Transformer-MAF.
  • B.3 MEAN SQUARED ERROR (MSE): Univariate methods typically outperform multivariate methods because estimating cross-series correlations is difficult.The passage notes that added multivariate variance can harm forecasts even when the series are related.
  • B.3 MEAN SQUARED ERROR (MSE): The proposed methods improve the metrics for the Traffic data set, and this pattern holds across the other experimental data sets.Prediction intervals are also plotted for visual comparison in Figures 8, 9, 10, and 11.

D.1 FEATURES

The experiments use normalized calendar features and frequency-specific lag inputs, while the reported figures visualize forecasts or prediction intervals against test-set ground truth across multivariate series.

  • D.1 FEATURES: Normalized calendar features are concatenated to the RNN or Transformer input at each time step, with choices determined by data frequency.Hourly data use hour-of-day, day-of-week, and day-of-month features; daily data use day-of-week features; minute-level data use minute-of-hour, hour-of-day, and day-of-week features.
  • D.1 FEATURES: Frequency-specific lag values are also concatenated as inputs, including [1, 24, 168] for hourly and [1, 7, 14] for daily data.
  • D.1 FEATURES: The figures show point forecasts or prediction intervals alongside test-set ground truth for selected Traffic and Electricity series.Traffic visualizations cover the first 16 of 963 series, while Electricity visualizations cover the first 16 of 370 series; the Traffic point-forecast figures report CRPSsum, CRPS, and MSE values for LSTNet and N-BEATS.
  • D.1 FEATURES: Training uses batches of 64, up to 40 epochs, learning rate 1e-3, and K = 5 stacks of normalizing-flow bijection layers.The flow components use linear feed-forward layers with hidden dimension 100 and ELU activations.
Loading 2002.06103v3…