PaperlayerView exact revision on arXiv

Source-linked AI summary

An Experimental Review on Deep Learning Architectures for Time Series Forecasting

Pedro Lara-Benítez, Manuel Carranza-García, José C. Riquelme

arXiv:2103.12057v2cs.LGcs.AI

TL;DR

Choosing and tuning deep learning architectures for time series forecasting is difficult, and existing reviews have not comprehensively compared the main architectures across forecasting tasks. This paper reviews the field and experimentally compares seven architectures across 12 problems, finding that LSTM is most accurate while CNN offers comparable performance with lower variability and greater efficiency.

  • Problem

    Existing reviews focus on particular architectures or data scenarios, leaving limited comprehensive evidence on which deep learning architectures suit different forecasting tasks.

  • Method

    The study reviews deep learning for forecasting and exhaustively compares seven architectures across 12 datasets using varied configurations and hyperparameters, evaluating accuracy and efficiency.

  • Results

    LSTM achieves the best WAPE results, while CNN outperforms recurrent models on mean and standard-deviation WAPE and provides the best speed/accuracy trade-off.

  • Takeaways & Limitations

    LSTM is the most accurate alternative, whereas CNN is more consistent and efficient, making it more suitable for real-time applications than recurrent approaches.

  • Takeaways & Limitations

    Elman recurrent networks can suffer from exploding gradients or excessive computational time when learning long-term patterns.

Abstract

from arXiv · show

In recent years, deep learning techniques have outperformed traditional models in many machine learning tasks. Deep neural networks have successfully been applied to address time series forecasting problems, which is a very important topic in data mining. They have proved to be an effective solution given their capacity to automatically learn the temporal dependencies present in time series. However, selecting the most convenient type of deep neural network and its parametrization is a complex task that requires considerable expertise. Therefore, there is a need for deeper studies on the suitability of all existing architectures for different forecasting tasks. In this work, we face two main challenges: a comprehensive review of the latest works using deep learning for time series forecasting; and an experimental study comparing the performance of the most popular architectures. The comparison involves a thorough analysis of seven types of deep learning models in terms of accuracy and efficiency. We evaluate the rankings and distribution of results obtained with the proposed models under many different architecture configurations and training hyperparameters. The datasets used comprise more than 50000 time series divided into 12 different forecasting problems. By training more than 38000 models on these data, we provide the most extensive deep learning study for time series forecasting. Among all studied models, the results show that long short-term memory (LSTM) and convolutional networks (CNN) are the best alternatives, with LSTMs obtaining the most accurate forecasts. CNNs achieve comparable performance with less variability of results under different parameter configurations, while also being more efficient.

1. Introduction

Time series forecasting is important across many real-life domains, while deep learning offers a way to model complex nonlinear interactions. This paper reviews deep learning for forecasting and benchmarks seven architectures across diverse datasets, configurations, and evaluation criteria.

  • Motivation: Time series forecasting supports real-life applications with temporal components, including weather, energy, finance, retail, and medical monitoring.Its goal is to predict future values by capturing temporal patterns and correlations among nearby observations.
  • Motivation: Deep learning has become a popular approach for time series forecasting because neural networks can represent complex nonlinear feature interactions beyond linear statistical models.Modern neural systems rely on deep structures that stack layers and densely connect many neurons.
  • Study scope: The study reviews existing deep learning techniques and compares seven model types: multi-layer perceptron, Elman recurrent, long short-term memory, echo state, gated recurrent unit, convolutional, and temporal convolutional networks.The review addresses prior work that focused on specific architectures or data scenarios.
  • Experimental design: The experiments evaluate accuracy, efficiency, and result distributions across 12 publicly available datasets from fields including finance, energy, traffic, and tourism.Different architecture and training hyperparameter configurations are examined, with 6432 models tested for each dataset.
  • Contribution: The work aims to establish a general and reliable deep-learning benchmark for future comparisons across many time series forecasting problems and domains.It assesses the most relevant neural-network types rather than comparing novel approaches only with classical models.
  • Contribution: The paper contributes an exhaustive review, a comparative analysis across datasets of different natures, and an open-source deep learning framework for time series forecasting.These contributions are summarized explicitly in the introduction.

2. Deep learning architectures for time series forecasting

Time series forecasting predicts future observations from past values, and this paper focuses on univariate series. It reviews deep learning architectures motivated by the limitations of traditional statistical models and the flexibility of neural networks for nonlinear relationships.

  • Forecasting problem: Time series forecasting predicts the next H observations from a series’ historical values and lagged observations.The historical data are represented as X = {x1, x2, ..., xT }, with forecasting horizon H.
  • Forecasting problem: The paper studies univariate time series, which contain a single observation recorded sequentially at each timestep.Time series may be univariate or multivariate, but this work considers only univariate analysis.
  • Traditional methods: Traditional TSF methods include ETS and ARIMA, which construct linear functions from recent observations to predict future values.These statistical models have been extensively used over the last decades.
  • Traditional methods: Statistical methods can fail when applied without stationarity, ergodicity, and preprocessing requirements.For example, ARIMA requires transforming the time series into a stationary form.
  • Neural architectures: Artificial neural networks can outperform statistical methods in many situations because their deep structures flexibly map nonlinear relationships from data.Researchers nevertheless face difficulty selecting optimal network topologies and learning algorithms because ANN architectures permit infinitely many configurations.
  • Neural architectures: The reviewed deep learning families include fully connected, recurrent, and convolutional networks, including MLP, LSTM, ERNN, ESN, GRU, CNN, and TCN variants.The paper discusses these architectural variants and their successful application fields in subsequent sections.

2.1. Multi-Layer Perceptron

MLP is the basic feed-forward neural-network architecture, consisting of input, hidden, and output layers with depth determined by the number of hidden layers. Studies found that input-history selection and preprocessing strongly influence forecasting accuracy, while MLPs cannot capture temporal order.

  • Architecture: MLP is the most basic feed-forward artificial neural network, comprising input, hidden, and output layers; one or multiple hidden layers determine network depth.The architecture was inspired by parallel processing among neurons in the brain.
  • Historical development: Early studies viewed MLPs as promising alternatives to statistical models but remained skeptical because methodologies were unsystematic and interpretation was difficult.These studies generally used one or a few hidden layers.
  • Historical development: Reviews identified ANNs as flexible universal function approximators for forecasting but called for more rigorous validation before claiming improvements over classical alternatives.Their flexibility allows adaptation to data without prior assumptions.
  • Empirical findings: Studies reported that the past-history parameter influences accuracy more than the number of hidden layers, while carefully selected input variables and preprocessing can improve simple models.Recent work also proposed ensembles, although the supplied passage truncates the associated conclusion.
  • Limitations: MLPs treat inputs independently and therefore cannot capture temporal order, limiting time-series forecasting performance, especially for dynamically changing instance lengths.This limitation motivated interest in recurrent and convolutional neural networks.

2.2. Recurrent Neural Networks

Recurrent neural networks model temporal dependencies by connecting each time step with previous observations, addressing a limitation of feed-forward networks. Subsequent architectures, including LSTM, ESN, and GRU, improve long-term memory, training efficiency, or computational time.

  • Recurrent Neural Networks: RNNs connect each time step with previous observations to model temporal dependencies and learn which past information is relevant to forecasting.Unlike MLPs, RNNs provide native support for sequence data.
  • Recurrent Neural Networks: TBTT can cause exploding gradients through weight oscillation or vanishing gradients that make learning long-term patterns computationally excessive.These are identified as two main problems in applying truncated backpropagation through time.
  • Long Short-Term Memory: LSTMs address ERNN limitations by modeling longer temporal horizons without forgetting short-term patterns through gated memory cells.Input and output gates protect memory from irrelevant perturbations, while a forget gate can reset irrelevant contents.
  • Echo State Networks: ESNs simplify training by tuning only output-neuron weights, making training a simple linear regression task rather than optimizing every network neuron.Their random, sparsely connected reservoirs act as echo functions that reproduce specific temporal patterns.
  • Echo State Networks: ESNs have outperformed MLPs and statistical methods on chaotic time series while remaining time-efficient because reservoir neurons are non-trainable.The passage compares ESN efficiency with other RNNs.
  • Gated Recurrent Units: GRUs simplify LSTM units by combining forget and input functions into one update gate, reducing trainable parameters and computational time while achieving similar results.GRUs also use a reset gate to determine how much past information to forget.

2.3. Convolutional Neural Networks

CNNs extract features from grid-structured raw data using local connectivity and shared weights, reducing trainable parameters and improving learning efficiency. TCNs adapt CNNs to temporal forecasting through causal convolutions, same-length sequence mapping, and efficient processing of long histories.

  • CNN fundamentals: CNNs automatically extract meaningful features from high-dimensional raw data with grid topology, without feature engineering.They were originally designed for computer vision and are state-of-the-art for several classification tasks.
  • CNN fundamentals: CNNs typically combine convolution, pooling, and fully connected layers based on local connectivity, shared weights, and translation equivariance.Pooling reduces feature-map spatial dimensions, while fully connected layers combine local features into global features.
  • CNN fundamentals: Shared convolutional filters and local receptive fields give CNNs fewer trainable parameters than RNNs, making learning more time efficient.Neurons connect only to input regions, and neurons in the same layer share the convolutional weight matrix.
  • CNNs for forecasting: CNN-based models remain less extensively used in time series forecasting than RNNs, although studies use them as feature extractors or with recurrent blocks.Relevant CNN forecasting studies are summarized in Table 6.
  • Temporal Convolutional Networks: TCNs use causal convolutions and map sequences of any length to outputs of the same length, enabling effective handling of large history sizes.Their reported advantages include low training memory, parallel processing of long sequences, and more stable training than sequential RNN processing.

3. Materials and methods

The study evaluates seven deep-learning architectures for time series forecasting across 12 diverse public datasets containing more than 50000 time series. It uses exhaustive architecture and training-parameter searches, WAPE-based accuracy and efficiency evaluation, and Friedman statistical analysis to compare models.

  • Datasets: 12 publicly available datasets span varied domains, sizes, time-series lengths, forecasting horizons, scales, and seasonality, totaling more than 50000 time series.The datasets include series from forecasting competitions and other time-series forecasting reviews.
  • Architectures and configurations: Seven model types are compared: multilayer perceptron, Elman recurrent, long short-term memory, echo state, gated recurrent unit, convolutional, and temporal convolutional networks.The study addresses the high number of hyperparameters by performing an exhaustive grid search for each architecture.
  • Architectures and configurations: The MLP baseline comprises 12 configurations varying hidden layers and units, with 8 to 128 neurons per layer to accommodate shorter and longer input histories.The designs also include encoder- and decoder-like structures with different layer-width patterns.
  • Training and preprocessing: Training uses min-max scaling and mean normalization, also known as z-score, with the complete training-parameter grid specified in Table 10.These are identified as the two most common normalization functions in the literature.
  • Evaluation procedure: Model performance is assessed for accuracy and efficiency using weighted absolute percentage error (WAPE), while the Friedman test compares global differences and ranks methods across datasets.The analysis considers both the best result for each architecture and the distribution of results across hyperparameter configurations.

4. Results and Discussion · 4.1. Forecasting accuracy

The experiments evaluate forecasting accuracy with WAPE across model architectures and datasets. Results indicate that LSTM and CNN models are consistently strong, while MLP performs worst overall and CNN often provides more stable outcomes.

  • 4. Results and Discussion: The experiments required around four months of computation using five NVIDIA TI-TAN Xp 12GB GPUs, Intel i7-8700 CPUs, and an additional Amazon cloud GPU.This passage reports the computational setting for the results and discussion.
  • 4.1. Forecasting accuracy: The study analyzes forecasting accuracy using the WAPE metric across distributions of model results for each dataset.Figure 3 summarizes means, quartiles, and overall result distributions.
  • 4.1. Forecasting accuracy: ERNN and TCN architectures are more sensitive to parametrization, showing wider WAPE distributions than CNN and MLP.The passage contrasts variability across architecture distributions.
  • 4.1. Forecasting accuracy: Except for MLP, most architectures can achieve accuracy close to the best model in almost all cases.Their minimum WAPE values are described as close to one another, although reaching that performance differs in difficulty.
  • 4.1. Forecasting accuracy: MLP models perform worst overall and serve as a simple comparison baseline for the other architectures.The passage attributes this result partly to MLPs not being specifically designed for time series.
  • 4.1. Forecasting accuracy: LSTM models achieve the best results in four out of twelve datasets and rank among the top three architectures on most remaining datasets.Tourism is identified as an exception, where LSTM ranks sixth.
  • 4.1. Forecasting accuracy: CNN outperforms the other models in mean and standard deviation of WAPE on many datasets.The complete architecture-configuration results and grouped summaries are provided in an online appendix.

4.2. Computation time

The architectures differ substantially in computational efficiency and in the trade-off between forecasting accuracy and computation time. MLP is fastest, while CNN offers the best overall time–accuracy balance and LSTM combines high predictive performance with adequate computation time.

  • MLP is the fastest architecture, closely followed by CNN, whose computation time differs significantly from the remaining models.
  • Among recurrent networks, LSTM and GRU have similar computation times, whereas ERNN is the slowest architecture overall.
  • TCN has average training and inference times comparable to GRU and LSTM, but its computation-time distribution is less variable.The results indicate that increasing depth has a smaller effect on computation time in convolutional than recurrent models.
  • CNN achieves the best time–accuracy balance, while MLP is fastest but least accurate, ERNN is accurate but resource-intensive, and TCN is significantly slower.Among recurrent networks, LSTM provides very high predictive performance with adequate computation time.

4.3. Statistical analysis

The statistical analysis compares deep learning architectures across forecasting accuracy, variability, computational efficiency, and aggregate rankings. LSTM provides the best forecasting accuracy, while CNN offers greater efficiency and lower result variability, making both suitable alternatives depending on problem requirements.

  • Forecasting accuracy: LSTM ranks first for best WAPE forecasting accuracy, closely followed by GRU, while differences among the best architectures are generally not statistically significant except for MLP.The rankings use Friedman tests and Holm-Bonferroni post-hoc critical-difference diagrams.
  • Forecasting accuracy: CNN ranks first for mean WAPE, with LSTM second; both are less sensitive to parametrization and occupy the top positions for WAPE standard deviation.Their ranking suggests they can reduce the cost of extensive grid searches.
  • Forecasting accuracy: ERNN provides the worst distribution of results, suggesting that its parametrization is complex and making it the least recommended method studied.Its distributional performance is reported as worse than MLP.
  • Computational efficiency: MLP ranks first in computational efficiency and has very low training-time standard deviation, while CNN is the most efficient deep learning model across all rankings.Training and inference time analyses produced similar conclusions, so the displayed rankings focus on training time.
  • Overall assessment: The aggregate ranking identifies CNN and LSTM as the most suitable alternatives: LSTM delivers the best forecasting accuracy, whereas CNN is more efficient and less variable.The preferred model depends on problem requirements, time constraints, and research objectives.

4.4. Model architecture configuration

The section evaluates architecture configurations by comparing best WAPE results across layers, units, filters, kernels, pooling, and sequence-return settings. Results show distinct design patterns for MLP, recurrent, CNN, and TCN models, including simpler configurations often performing better.

  • Configuration analysis: Configurations are ordered by WAPE average rankings, with only the best result reported after repeated training under different hyperparameters.Rankings are independent for each of the seven network types.
  • MLP configurations: The top-ranked MLP uses 5 hidden layers and 80 total neurons, while increasing hidden-neuron count does not guarantee better performance.The most complex 320-neuron MLP distributed across 5 layers ranks penultimate.
  • Recurrent configurations: LSTM models achieve the best recurrent performance with two stacked layers of 32 units returning the complete sequence before the output dense layer.Across recurrent architectures, smaller unit counts generally perform better, while increasing layers worsens average results for LSTM, ERNN, and GRU.
  • CNN configurations: CNN accuracy increases with convolutional depth, with four-layer models ranking best and single-layer models ranking lowest.Filter count shows no clear difference, and the best predictions come from models without max-pooling.
  • TCN configurations: The best TCN has one TCN layer containing 4 dilated convolutional layers, kernel length 6, and 64 filters without complete sequence return.Single-layer models with few dilated layers outperform more-complex models, while larger kernels provide better predictions.
  • TCN configurations: Returning the complete sequence before the final dense layer can increase model complexity excessively and reduce TCN performance.This pattern is reported specifically for the TCN configurations.

4.5. Statistical comparison of architecture configurations

A paired Wilcoxon signed-rank analysis identifies statistically preferred architecture configurations and confirms that parameter effects vary across model types. Recurrent models generally favor simpler configurations, while CNNs and TCNs show distinct layer, pooling, and kernel-size preferences.

  • Statistical method and interpretation: The study compares architecture configurations with paired Wilcoxon signed-rank tests to validate previously identified parameter choices.Results classify choices as significantly better at p < 0.05, tendencies at p < 0.2, or indistinguishable when no significant differences are found.
  • Recurrent networks: ESN performs significantly better when returning the whole sequence, ERNN tends to work better without returned sequences, and LSTM and GRU are unaffected.The test also finds that LSTM parameter choices have minimal impact on accuracy, supporting their robustness.
  • CNN and TCN models: CNNs perform significantly better with four stacked layers than with one or two, while filter count has no effect and max-pooling is not recommended.These findings identify depth as beneficial for CNNs but exclude max-pooling for the evaluated time-series forecasting problems.
  • CNN and TCN models: Single-layer TCNs outperform more-complex models, and larger kernel sizes provide significantly better predictions.The statistical comparison confirms that TCN performance favors simpler depth and larger receptive fields.

4.6. Analysis of the training parameters

Across architectures, the best training configurations generally use a low learning rate and small past-history factor, while normalization and batch-size preferences vary by model.

  • Training hyperparameters: Low learning rates and small past-history factors produce the best results across all architectures.These patterns were identified from the grid search over training hyperparameters.
  • Normalization: Minmax normalization performs better for GRU and LSTM, whereas z-score significantly outperforms minmax in the other architectures.Normalization preferences therefore depend on the deep learning architecture.
  • Batch size: Most architectures—ERNN, GRU, CNN, and TCN—perform better with larger batch sizes, while LSTM prefers a batch size of 32.The passage contrasts the larger-batch preference of most listed architectures with LSTM’s preferred value.

5. Conclusions

The study experimentally reviewed seven deep learning architectures across 12 forecasting problems and more than 50000 time series. Correct parametrization was important; LSTM achieved the best WAPE results, while future work targets streaming, data characteristics, regularization, multivariate series, ensembles, transfer learning, and benchmarking.

  • Experimental scope: The study evaluated seven deep learning architectures for accuracy and efficiency across 12 forecasting problems containing more than 50000 time series.The architectures were MLP, ERNN, LSTM, GRU, ESN, CNN, and TCN.
  • Main conclusions: Except MLP, all studied models produced accurate predictions when parametrized correctly, but their result distributions differed substantially.The findings emphasize the importance of identifying an optimal architecture configuration.
  • Main conclusions: Regardless of hidden-block depth, MLP networks could not model temporal order and therefore showed poor predictive performance.The conclusion specifically attributes MLP’s weakness to its inability to represent temporal ordering in the data.
  • Main conclusions: LSTM obtained the best WAPE results, followed by GRU.The supplied conclusion passage reports this ranking but is truncated before completing the comparison of the remaining models.
  • Future work: Future work will examine forecasting in streaming scenarios, speed-versus-accuracy trade-offs, data characteristics, regularization, multivariate series, ensembles, transfer learning, and larger benchmark databases.These extensions address real-time forecasting, model selection, training cost, accuracy enhancement, and validation of new proposals.
Loading 2103.12057v2…