Time Series Forecasting With Deep Learning: A Survey

Bryan Lim; Stefan Zohren

Source-linked AI summary

Time Series Forecasting With Deep Learning: A Survey

Bryan Lim, Stefan Zohren

arXiv:2004.13408v2Apr 28, 2020stat.MLcs.LG

TL;DR

Time series forecasting spans diverse datasets and requires architectures that incorporate temporal information for one-step and multi-horizon predictions. The paper surveys encoder-decoder designs, hybrid statistical-neural models, and extensions for decision support. It concludes that these approaches broaden forecasting and decision-support capabilities, while continuous-time and hierarchical settings remain important limitations and future directions.

Problem
Time series forecasting requires neural architectures that accommodate diverse datasets and incorporate temporal information across one-step-ahead and multi-horizon prediction settings.
Method
The paper surveys encoder and decoder designs, hybrid models combining statistical and deep learning components, and neural-network extensions for interpretability and causal inference over time.
Results
The survey finds that hybrid deep learning models can outperform pure statistical or machine learning methods, and that attention can identify important time points and persistent temporal relationships.
Takeaways & Limitations
Deep learning for time series can support both forecasting and user decision-making through temporal modelling, interpretability, and counterfactual prediction.
Takeaways & Limitations
Deep learning models typically require regularly discretised observations, making irregular or missing observations difficult to forecast; hierarchical time-series structures also require further work.

Abstract

from arXiv · show

Numerous deep learning architectures have been developed to accommodate the diversity of time series datasets across different domains. In this article, we survey common encoder and decoder designs used in both one-step-ahead and multi-horizon time series forecasting -- describing how temporal information is incorporated into predictions by each model. Next, we highlight recent developments in hybrid deep learning models, which combine well-studied statistical models with neural network components to improve pure methods in either category. Lastly, we outline some ways in which deep learning can also facilitate decision support with time series data.

1. Introduction

Time series modelling supports applications across many domains, while deep learning offers data-driven ways to learn complex temporal representations. This survey organizes modern forecasting architectures, hybrid models, decision-support methods, and future directions, while acknowledging that its coverage is not exhaustive.

Time series modelling underpins applications in climate, biology, medicine, retail, and finance.
Traditional approaches use domain-informed parametric models, whereas machine learning can learn temporal dynamics in a data-driven manner.
Deep neural networks use architectural inductive biases to learn complex representations and reduce reliance on manual feature engineering and model design.
The survey covers forecasting techniques, hybrid statistical-neural models, neural-network approaches to interpretability and counterfactual prediction, and future continuous-time and hierarchical models.
The survey is not all-encompassing, omitting or giving limited coverage to automated parametric selection, traditional machine learning, Gaussian processes, and older neural-network models.

2. Deep Learning Architectures for Time Series Forecasting

Time series forecasting predicts an entity’s future target value from recent target observations, exogenous inputs, and static metadata. The simplest one-step-ahead formulation learns a prediction function over a look-back window.

Forecasting models predict the future target value for each entity, such as a weather station’s measurement or a patient’s vital sign.
A one-step-ahead forecast uses target and exogenous observations from a look-back window together with entity-specific static metadata.
The prediction function maps these temporal observations and static metadata to the model forecast.
Although the survey focuses on univariate targets, the same components can extend to multivariate models without loss of generality.

(a) Basic Building Blocks

Deep forecasting architectures encode historical information into latent representations using different temporal mechanisms. The survey contrasts convolutional, recurrent, and attention-based encoders and relates these choices to the temporal relationships they can learn.

General encoder-decoder framework: Encoders transform historical observations into a latent variable used to produce the final forecast.Encoder and decoder functions form the basic building blocks of deep learning architectures.
(i) Convolutional Neural Networks: Causal CNNs apply filters using only past information, but their learned relationships are time-invariant and restricted to a finite receptive field.The receptive field determines which historical inputs a CNN can use for forecasting.
(i) Convolutional Neural Networks: Dilated convolutions reduce the resolution of lower-layer features so successive layers can aggregate distant history more efficiently.Increasing dilation rates across layers allows information from different time blocks to be incorporated.
(ii) Recurrent Neural Networks: RNNs recursively update an internal memory state, providing a compact summary of past information without explicitly specifying a look-back window.The recurrent layer resembles a nonlinear infinite impulse response filter from a signal-processing perspective.
(ii) Recurrent Neural Networks: LSTMs improve long-range dependency learning by using a cell state and gates to regulate information flow and improve gradient flow.The input, output, and forget gates modify the hidden and cell states.
(iii) Attention Mechanisms: Attention aggregates temporal features with dynamically generated weights, allowing models to focus directly on significant or distant time steps.Attention uses keys, queries, and values produced at different time steps, with the resulting context vector contributing to the encoded representation.
(iii) Attention Mechanisms: Attention-based forecasting models have shown improved performance over comparable recurrent networks and can learn event-specific and regime-specific temporal patterns.Examples include attending to holidays or promotions in retail and using distinct weight patterns for different regimes.

(b) Multi-horizon Forecasting Models

Multi-horizon forecasting extends one-step prediction across a discrete future horizon and uses iterative or direct architectures to generate forecasts. Iterative models reuse autoregressive predictions, whereas direct models use all available inputs through sequence-to-sequence or fixed-length outputs.

Multi-horizon forecasting provides predictions at multiple future points so decision makers can visualize trends and optimize actions across the entire path.
Forecast horizons use known future inputs across the horizon alongside inputs observed only historically.
Iterative Methods: Iterative models recursively feed sampled targets into future steps and generate forecasts from sampling distributions at each horizon.
Iterative Methods: Iterative models generalize one-step architectures easily, but small per-step errors can accumulate over longer horizons and observed inputs may be unavailable at run time.
Direct Methods: Direct methods use sequence-to-sequence architectures that encode past targets and inputs, then decode them with known future inputs; simpler variants directly output a fixed-length horizon.

3. Incorporating Domain Knowledge with Hybrid Models

Hybrid forecasting models combine quantitative time-series structure with deep learning components to address limitations of pure statistical and machine-learning approaches. They encode parameters or modify forecasting equations, separating broad temporal structure from learned effects and supporting probabilistic prediction.

Hybrid models combine statistical time-series models with deep learning and have demonstrated improved performance over pure statistical or machine-learning models across applications.
Hybrid models let domain experts inform neural-network training, reducing the hypothesis space and improving generalization, especially on small datasets.
Neural networks encode time-varying parameters for non-probabilistic models or produce distribution parameters for probabilistic models.
Non-probabilistic hybrid models: Non-probabilistic hybrids modify analytical forecasting equations; ES-RNN combines Holt-Winters level and seasonality components with deep-learning outputs.
Non-probabilistic hybrid models: Exponential-smoothing components capture broader trends and reduce the need for additional input scaling.
Probabilistic hybrid models: Probabilistic hybrids use neural networks to produce predictive-distribution parameters, including for Gaussian-process and linear state-space temporal models.

4. Facilitating Decision Support Using Deep Neural Networks

Deep learning can support time-series decisions by making forecasts more interpretable and enabling counterfactual scenario analysis. The surveyed approaches address both understanding model predictions and estimating outcomes under alternative actions despite time-dependent confounding.

Forecasts guide actions such as prioritizing medical tests, formulating diagnoses, and determining treatment courses.
Mission-critical neural-network deployment increases the need to understand both how and why models make predictions.
Post-hoc interpretability: Post-hoc interpretability applies simpler surrogate models between neural-network inputs and outputs to explain trained networks without modifying their original weights.
Inherent interpretability: Attention-based architectures provide inherent interpretability because attention weights indicate the relative importance of features at each time step.
Inherent interpretability: Attention-vector distributions can identify persistent temporal relationships such as seasonal patterns.
Counterfactual prediction: Counterfactual forecasts support scenario analysis by estimating how different actions can affect target trajectories.
Counterfactual prediction: Time-dependent confounding creates circular dependencies when actions affecting targets are conditioned on target observations, motivating deep-learning approaches that jointly model target and action distributions.

5. Conclusions and Future Directions

The survey reviews deep-learning architectures for one-step and multi-horizon forecasting, hybrid statistical-neural models, and decision-support methods based on interpretability and counterfactual prediction. Future work is needed for irregular observations and hierarchical time-series structures.

The survey examines encoder-decoder building blocks, their temporal information mechanisms, and extensions from one-step-ahead to multi-horizon forecasting.
Hybrid deep-learning models combine statistical and neural components to outperform pure methods in either category.
Interpretability and counterfactual prediction extend deep learning to improve decision support over time.
Future directions: Regular-interval discretization makes standard deep neural networks difficult to apply when observations are missing or arrive at random intervals.
Future directions: Continuous-time models need extension to complex inputs and benchmarking against existing models.
Future directions: Architectures that explicitly model logical hierarchies between trajectories could improve forecasting beyond existing univariate or multivariate models.