PaperlayerView exact revision on arXiv

Source-linked AI summary

Deep Learning for Time Series Forecasting: Tutorial and Literature Survey

Konstantinos Benidis, Syama Sundar Rangapuram, Valentin Flunkert, Yuyang Wang, Danielle Maddix, Caner Turkmen, Jan Gasthaus, Michael Bohlke-Schneider, David Salinas, Lorenzo Stella, Francois-Xavier Aubet, Laurent Callot, Tim Januschowski

arXiv:2004.10240v2cs.LGstat.ML

TL;DR

Deep forecasting addresses time-series problems involving many related series and complex patterns, where conventional approaches may be limited. The article presents a tutorial and literature survey organized around neural-network building blocks and recent model developments. It concludes that these methods are practically valuable but not universally appropriate, with important challenges remaining around data effectiveness, interpretability, explainability, and causality.

  • Problem

    Forecasting increasingly involves large panels of related series and complex patterns, motivating understanding and improving neural-network methods for these settings.

  • Method

    The article teaches core deep-forecasting concepts and surveys recent neural-network forecasting models through practical building blocks and archetypal architectures.

  • Results

    Deep forecasting methods show strong practical performance, including state-of-the-art results in M4 and M5 contexts and benefits for groups of similar or heterogeneous series.

  • Takeaways & Limitations

    Global neural-network models can support automated forecasting across groups of related series, while hybrid methods and innovations from adjacent areas remain promising directions.

  • Takeaways & Limitations

    Neural networks are not appropriate for every forecasting problem, particularly long-range macroeconomic forecasts or settings requiring external domain knowledge unavailable from the data.

Abstract

from arXiv · show

Deep learning based forecasting methods have become the methods of choice in many applications of time series prediction or forecasting often outperforming other approaches. Consequently, over the last years, these methods are now ubiquitous in large-scale industrial forecasting applications and have consistently ranked among the best entries in forecasting competitions (e.g., M4 and M5). This practical success has further increased the academic interest to understand and improve deep forecasting methods. In this article we provide an introduction and overview of the field: We present important building blocks for deep forecasting in some depth; using these building blocks, we then survey the breadth of the recent deep forecasting literature.

1 Introduction

Forecasting addresses future extrapolation across increasingly large panels of related series, creating opportunities for neural networks despite their historically mixed effectiveness. This article focuses on recent deep forecasting developments and combines an educational tutorial with a literature overview organized around practical model-building components.

  • Forecasting extrapolates time series into the future and supports applications including retail demand, traffic, energy, and finance.
  • Modern forecasting often involves forecasting large panels of related time series simultaneously, making these problems amenable to deep learning.
  • Neural networks historically had mixed forecasting effectiveness, although breakthroughs and advances gradually expanded their capabilities.
  • Recent deep forecasting methods gained practical and academic attention through industrial applications and strong M4 and M5 competition results.
  • The article prioritizes recent developments, sacrificing historical comprehensiveness for recency while targeting a general audience.
  • Its two-part structure provides a deep forecasting tutorial followed by an overview of modern models organized around their main building blocks.

2 Deep Forecasting: A Tutorial

The tutorial formalizes probabilistic multi-step forecasting and organizes deep forecasting around model archetypes and reusable neural-network building blocks. It emphasizes composing architectures, input transformations, and output representations while accounting for forecasting-specific constraints.

  • 2.1 Notation and Formalization of the Forecasting Problem: Forecasting is framed as estimating a conditional distribution over future values for potentially multivariate series and multiple forecast steps.The formulation accommodates multidimensional series, multiple time series, and an h-step horizon.
  • 2.1 Notation and Formalization of the Forecasting Problem: Local univariate models train independently per series, preventing information sharing and making them unsuitable for cold-start forecasting.Each model predicts only one series, even when covariates are multidimensional.
  • 2.1 Notation and Formalization of the Forecasting Problem: Global models share parameters across series, pooling information while still producing a univariate forecast for one series at a time.Shared parameters can improve inference for shorter series relative to local univariate models.
  • 2.1 Notation and Formalization of the Forecasting Problem: Multivariate models directly predict all series jointly and learn their dependency structure.This distinguishes global multivariate models from local multivariate approaches such as VARMA.
  • 2.2.1 Multilayer perceptron: MLPs stack fully connected neuron layers but do not exploit data structure and require fixed input and output sizes, limiting their forecasting applicability.These limitations motivate architectures such as CNNs and RNNs that better accommodate structured or varying-size inputs and outputs.
  • 2.2.2 Convolutional neural networks: CNNs exploit ordinal structure through local convolutions, while causal and dilated convolutions restrict inputs to the past and expand the receptive field.Attention mechanisms instead learn which input representations are useful for each prediction, particularly for long sequences.
  • 2.3 Input Transformations: Global forecasting requires carefully scaled inputs, with normalization computed from past windows to avoid leakage from future values.Input transformations such as Box-Cox can be integrated into training through an optimized free parameter.

2.4 Output Models and Loss Functions

Deep forecasting models can output point estimates or richer representations of predictive distributions, including PDFs, quantiles, spline-based quantile functions, and normalizing flows. The section also connects these output choices to forecasting losses and architectures such as DeepAR and sequence-to-sequence models.

  • Output representations: Point forecasts output a single estimate optimized for a chosen loss, while probabilistic forecasts represent uncertainty through a predictive distribution.Probabilistic forecasts are especially relevant when downstream decisions minimize expected cost.
  • Probability distributions: Parametric probabilistic outputs can use Gaussian, Student-t, Tweedie, negative binomial, or mixture distributions depending on the data.Neural networks may output distribution parameters such as the mean and variance of a Gaussian.
  • Quantile functions: Quantile regression estimates selected quantiles with pinball loss, with q = 0.5 corresponding to the median point forecast.Estimating quantiles directly can target one quantile or multiple quantiles simultaneously.
  • Quantile functions: Linear isotonic splines provide a constrained quantile-function parameterization, trained directly with CRPS, while adaptive binning corresponds to piecewise linear CDFs and piecewise constant PDFs.The spline domain is restricted to [0, 1] and monotonicity is enforced through reparameterization.
  • Density estimation: Normalizing flows use invertible neural networks to transform simple distributions into complex ones while preserving probability mass and enabling density evaluation.The change-of-variables formula combines the transformed base density with the absolute Jacobian determinant.
  • Forecasting architectures: DeepAR uses an RNN with lagged targets and covariates to produce point forecasts or distribution parameters, while sequence-to-sequence models output multiple future targets directly.DeepAR’s basic probabilistic variant uses a PDF and trains with negative log-likelihood.

3 Literature review

The literature review builds on the tutorial’s forecasting foundations to survey modern deep forecasting models across probabilistic, point, hybrid, multivariate, physics-based, global-local, intermittent, and adversarial settings. Because the literature is broad, the selection is explicitly subjective.

  • Scope: The authors state that their literature selection is necessarily subjective because of the breadth of available deep forecasting research.The survey prioritizes coverage of modern developments rather than an exhaustive catalog.
  • Scope: The survey covers probabilistic and point forecast models, along with hybrids, multivariate models, physics-based models, global-local models, intermittent series, and GANs.It concludes with an overview of available models and guidance on where to begin.
  • Organization: The review uses the tutorial’s recipes to relate one-step and multi-step forecasting, and point forecasting to probabilistic forecasting.These transformations are discussed after the introductory material in Section 2.

3.1 Probabilistic Forecast Models

Probabilistic forecasting research extends canonical models with richer distributions, multivariate dependence, quantile methods, alternative convolutional backbones, and encoder-decoder structures. The surveyed approaches differ in how they generate future values and represent uncertainty.

  • Canonical models: DeepAR is a canonical one-step global univariate model that uses an RNN and can output PDFs, Gaussian mixtures, or quantile functions.Its input combines lagged targets with relevant covariates.
  • Canonical models: TimeGrad combines an LSTM or GRU with a diffusion probabilistic model to scale to multivariate series and use dependencies between dimensions.Samples are drawn from the data distribution at each time step, as in DeepAR.
  • Convolutional models: Convolutional alternatives replace recurrent backbones with dilated causal convolutions, while WaveNet adaptations retain efficient GPU training but sequential prediction.WaveNet was originally introduced as a generative speech-synthesis architecture.
  • Sequence-to-sequence models: Sequence-to-sequence architectures output all future target values at once, avoiding the need to unroll one-step forecasts across the horizon.MQRNN and MQCNN are presented as probabilistic seq2seq examples.
  • Quantile models: Generative quantile-copula models make MQCNN generative by learning conditional marginal quantiles and combining them with a Gaussian copula for correlated samples.The quantile index is sampled from a uniform distribution during training.
  • Quantile models: Incremental quantile functions provide flexible distribution-free quantile estimation, resolve quantile crossing with a neural-network layer, and can use CRPS training.The framework uses a seq2seq encoder-decoder structure but can also apply to recurrent one-step forecasts.

3.2 Point Forecast Models

Point forecasting models directly predict summary statistics rather than full future distributions, using recurrent, convolutional, autoencoder, residual, attention, and Transformer-based architectures. The survey also highlights that model design and preprocessing interact with the relatedness and seasonal structure of time-series panels.

  • Foundations: Point forecast models directly output a summary statistic of future values instead of modeling the full predictive distribution.Generic recipes can convert point forecasting models into probabilistic models.
  • One-step forecast: LSTNet combines convolutional layers, recurrent layers, temporal attention, and an autoregressive component for multivariate one-step point forecasts.Its output can be adapted to probabilistic forecasting by predicting distribution parameters and using NLL.
  • One-step forecast: N-BEATS uses deep residual stacks of forked MLPs to learn backcast and forecast components for point prediction.The residual architecture removes explainable context before subsequent processing.
  • One-step forecast: Stacked autoencoders learn features from spatio-temporal traffic data before a logistic regression layer predicts future flows at all locations.The architecture is trained layer-wise in a greedy manner and improves over shallow architectures in reported experiments.
  • Global-local models: Clustering disparate time series before training one RNN per cluster addresses cases where a single global model may not capture the series well.The approach uses feature-based clustering and is applied to public competition datasets.
  • Global-local models: Seasonality decomposition helps global models for somewhat unrelated panels but may harm performance for related or homogeneous panels.The decomposition steps are not integrated into the neural network for end-to-end learning.
  • Transformer models: Forecasting Transformers can add causal convolutions and sparse attention, reducing memory complexity from quadratic to O(T(log T)^2) for long sequences.The Temporal Fusion Transformer adds static-covariate embeddings, variable selection, gating, and quantile prediction.

3.3 Deep State Space Models

Deep state space models combine neural networks with classical state space models to learn shared patterns while retaining probabilistic forecasting structure. Recent extensions relax Gaussian, linear-dynamics, and fixed-switching assumptions, trading exact tractability for greater flexibility in some cases.

  • Hybrid state space models: Rangapuram et al. combine classical state space models with deep learning to bridge structural assumptions and cross-series pattern learning.The shared parameters are learned by maximizing likelihood over observations.
  • Hybrid state space models: After learning shared parameters, forecasts are obtained from the state space model's future parameters and likelihood-based inference.For linear-Gaussian models, likelihood terms can be computed efficiently with Kalman filtering.
  • Relaxing distributional assumptions: Normalizing Kalman Filters address Gaussian-observation limitations by transforming non-Gaussian, multimodal observations into more Gaussian-like data with learnable flows.They retain exact inference and likelihood computation, efficient sampling, and robustness to noise from linear Gaussian state space models.
  • Relaxing model assumptions: Kurle et al. address both Gaussian-observation and linear-dynamics limitations using switching linear dynamics and a nonlinear decoder emission model.Inference is no longer fully tractable, but Gaussian latent-state inference and likelihood estimation can be Rao-Blackwellised.
  • Switching dynamics: RED-SDS extends state space models with state-dependent switching and explicit duration variables to model state- and time-dependent dynamics.The method uses a recurrent state-to-switch connection and an explicit duration count variable.

3.4 Multivariate Forecasting

Multivariate forecasting models exploit dependencies among series rather than forecasting each dimension independently. Approaches include copulas, hierarchical coherence, and graph-based inputs for interpretable relational structures such as traffic networks.

  • Multivariate structure: Multivariate forecasting generalizes univariate models by exploiting dependency structures among the different series in a panel.This is especially relevant when multiple dimensions must be forecast simultaneously.
  • Copula models: RNN-based copula models represent multivariate dependence with either nonparametric or Gaussian copulas.Salinas et al. use a low-rank covariance approximation to scale the Gaussian copula process to thousands of dimensions.
  • Hierarchical forecasting: Hierarchical forecasting manages a trade-off between forecast accuracy and coherence with a predefined aggregation structure.Coherent forecasts conform to the hierarchy so aggregated forecasts equal forecasts of aggregates.
  • Graph-based models: Graph neural networks can encode interpretable connections between dimensions in the input layer, such as neighboring locations in city traffic networks.PEMS-BAY and METR-LA are examples of traffic-network forecasting settings.

3.5 Physics-based Models

Physics-based deep forecasting models incorporate mechanistic dynamics through differential equations and hybrid architectures. Neural ODE extensions address time-varying parameters, while AutoODE targets distribution shifts that challenge deep sequence models.

  • Neural ODEs: Neural ODE models solve an ordinary differential equation forward in time and the adjoint equation backward using backpropagation.They provide a physics-based way to model underlying dynamics.
  • Neural ODEs: A Neural ODE limitation is that its unknown parameters θ are assumed constant in time.Vialard et al. introduce a shooting formulation that allows parameters θ(t) to vary over time.
  • Hybrid dynamics: Benchmarking on COVID-19 and Lotka–Volterra population dynamics shows that distribution shifts can pose problems for deep sequence models.The study proposes the hybrid AutoODE model to represent underlying dynamics.
  • Neural ODE connections: Residual networks can be expressed as Forward Euler discretizations of ordinary differential equations with time step Δt = 1.This connects a common deep architecture to an ODE-based formulation.

3.6 Global-local

Global-local models combine information shared across many series with series-specific structure. They seek to retain the cross-series learning of global models while preserving local data efficiency, uncertainty estimation, or adaptive components.

  • Local and global models: Local models learn parameters separately for each series and provide uncertainty estimates, but cannot effectively share information or solve cold-start cases.Classical examples include state space models, ARIMA, exponential smoothing, and Gaussian processes.
  • Local and global models: Global models learn parameters jointly across a collection and can extract patterns from irregular series that single-series models may not distinguish.Neural networks have been particularly well suited to this global role.
  • Global-local combinations: Global-local models combine fixed global effects with random local effects to represent population-wide structure and individual or subgroup idiosyncrasies.Mixed-effect and hierarchical Bayesian models are examples of this combination.
  • Global-local architectures: Deep Factors represent each series as a weighted sum of a global time series and a local model.The approach combines neural-network expressiveness with classical models' data efficiency and uncertainty estimation.
  • Global-local architectures: DeepGLO forecasts collections of up to millions of series by combining global matrix factorization with a temporal deep leveled network.Its temporal convolution handles different scales without normalization or rescaling.
  • Global-local architectures: ES-RNN won the M4 competition by a large margin on both evaluation settings using local exponential-smoothing components with a global RNN.Its forecasts multiplicatively combine local level and seasonal components with the RNN output.

3.7 Intermittent Time Series

Research on intermittent time series uses neural networks to model sparse demand, extending classical decompositions and addressing limitations of point forecasts.

  • Neural networks have outperformed classical models by a significant margin in an early intermittent-demand comparison.Gutierrez et al. compared a shallow, narrow MLP with Croston’s method.
  • Kourentzes proposed shallow MLPs that use demand sizes and intervals as inputs, with alternative output designs for interval timing and occurrences.The two architectures differ in whether interval times and non-zero occurrences are output separately or combined as a ratio.
  • Kourentzes’s approach improved inventory metrics but not forecasting-accuracy metrics, challenging the earlier comparison.The source also notes uncertainty about whether the models were global or local.
  • Renewal-process approaches use RNNs to modulate discrete- and continuous-time processes, replacing exponential smoothing in Croston-like modeling.These methods address the point-forecast limitation of earlier neural approaches.
  • Point-process models combine asynchronous event sequences with uniformly sampled time-series observations for forecasting enriched with discrete events.Xiao et al. parameterize the probabilistic point-process model with two RNNs.

3.8 Generalized Adversarial Networks

GANs are used in time-series research for data synthesis and adversarial forecasting objectives, but their forecasting use remains comparatively limited.

  • GANs have received less attention in forecasting than in deep learning overall, possibly because CRPS can evaluate generated samples without a discriminator.The source presents this explanation as speculation.
  • Recent time-series GANs synthesize data or apply adversarial losses to forecasting tasks, often using recurrent networks for temporal dynamics.The literature includes several recurrent architectures for time-series GANs.
  • Modeling long-range dependencies and scaling recurrent networks to longer sequences limits time-series GAN applications to short sequences.The passage identifies this as an inherent difficulty of recurrent architectures in this setting.
  • Self-attention can model long-range dependencies from convolutional feature maps and has been successful in time-series forecasting.Convolutional architectures learn local features, while self-attention bridges their receptive-field limitation.

3.9 Summary and Practical Guidelines

The survey organizes modern deep forecasting methods and offers practical selection guidance, while emphasizing that no architecture is consistently superior across problems.

  • Model summary: Table 2 summarizes modern deep forecasting models by structure, forecast type, horizon, loss, metrics, data types, and comments.The table distinguishes probabilistic from point forecasts and one-step from direct multi-step horizons.
  • Model selection: No deep forecasting architecture has yet shown clearly superior empirical effectiveness across problems.The survey contrasts this with Transformer dominance in natural language processing and tree-based dominance in some forecasting settings.
  • Model selection: The authors provide non-exhaustive, experience-based guidelines to make architecture exploration more informed.The guidance is intended as a starting point rather than a complete selection procedure.
  • Baseline methods: DeepAR and MQCNN are recommended as baseline starting points, with N-BEATS, WaveNet, and Transformer-based models as additional candidates.The recommendation reflects the authors’ view of DeepAR and MQCNN as mature deep forecasting models.
  • Baseline methods: Global neural models are suggested when enough data are available, including for heterogeneous groups where recent competitions show state-of-the-art performance.The M4 and M5 competitions are cited as evidence that top-performing models had some form of globality.
  • Data characteristics: Reported neural-network advantages begin at 50000 observations for demand forecasting and a few hundred observations for load forecasting.The survey states that the relevant sample-size limitations remain incompletely understood theoretically and empirically.
  • Problem characteristics: Autoregressive models perform better than state-space models for forecast accuracy especially with less noisy data and shorter horizons.State-space models are described as robust to missing or noisy observations and potentially stronger when constraints are encoded in latent states.
  • Practical constraints: Computational constraints, inference latency, and available expertise can favor simpler neural architectures during model exploration.The authors also suggest considering CNNs versus RNNs in light of practical constraints and team skills.

4 Conclusions and Avenues for Future Work

The article introduces and surveys neural-network-based forecasting, while emphasizing that deep forecasting is useful but not universally appropriate. It identifies unresolved challenges, future methodological directions, and expanding application areas.

  • The article provides an introduction and overview of neural networks for forecasting, then reviews recent advances in deep forecasting models.
  • Deep forecasting excels with many related time series and large datasets, but is less appropriate for long-range macroeconomic forecasts or problems requiring external domain knowledge.
  • Many challenges remain, including data effectiveness, interpretability, explainability, causality, and the development of hybrid methods and new neural-network architectures.
  • Deep reinforcement learning may enable forecasting systems to learn optimal business decisions directly rather than supplying forecasts to downstream optimization problems.
  • Forecasting methods may find further applications in database management, cloud computing, system operations, and hyperparameter optimization.
Loading 2004.10240v2…