PaperlayerView exact revision on arXiv

Source-linked AI summary

Recurrent Neural Networks for Time Series Forecasting: Current Status and Future Directions

Hansika Hewamalage, Christoph Bergmeir, Kasun Bandara

arXiv:1909.00590v5cs.LGcs.NEstat.ML

TL;DR

The paper addresses whether RNNs can become practical forecasting tools despite concerns about their competitiveness, complexity, and usability relative to traditional statistical methods. It reviews and evaluates RNN forecasting architectures, finding competitive performance across many datasets and identifying when deseasonalization is needed.

  • Problem

    RNN forecasting faced skepticism because earlier evaluations often favored simpler statistical methods, while complex architectures lacked convincing empirical comparisons.

  • Method

    The paper combines a literature review, an open-source RNN software framework, and extensive empirical evaluations of standard architectures for univariate forecasting across diverse datasets.

  • Results

    RNNs can be competitive with ETS and auto.arima, with deseasonalization recommended unless datasets contain sufficiently long, homogeneous seasonal patterns.

  • Takeaways & Limitations

    RNNs are viable forecasting options for practitioners, particularly when global models exploit information across many related time series.

  • Takeaways & Limitations

    RNNs are typically computationally more expensive than traditional univariate forecasting techniques.

Abstract

from arXiv · show

Recurrent Neural Networks (RNN) have become competitive forecasting methods, as most notably shown in the winning method of the recent M4 competition. However, established statistical models such as ETS and ARIMA gain their popularity not only from their high accuracy, but they are also suitable for non-expert users as they are robust, efficient, and automatic. In these areas, RNNs have still a long way to go. We present an extensive empirical study and an open-source software framework of existing RNN architectures for forecasting, that allow us to develop guidelines and best practices for their use. For example, we conclude that RNNs are capable of modelling seasonality directly if the series in the dataset possess homogeneous seasonal patterns, otherwise we recommend a deseasonalization step. Comparisons against ETS and ARIMA demonstrate that the implemented (semi-)automatic RNN models are no silver bullets, but they are competitive alternatives in many situations.

1. Introduction

The paper addresses skepticism about neural-network forecasting by systematically evaluating practical RNN architectures against established statistical benchmarks and developing a standard software framework. It focuses on whether off-the-shelf RNNs can provide competitive, usable alternatives for practitioners, especially when many related time series are available.

  • Motivation: Past forecasting competitions often found neural networks less competitive than simpler univariate statistical methods.This contributed to neural networks being labelled unsuitable for forecasting.
  • Motivation: Short or changing time series can limit complex neural-network models, while their black-box nature has encouraged practitioners to prefer straightforward statistical techniques.Complex approaches require adequately long series generated by comparatively stable systems.
  • Motivation: Many related time series create a setting where isolated univariate methods may fail, whereas neural networks can benefit from massive datasets.Here, Big Data refers primarily to many related series rather than abundant observations in each individual series.
  • Motivation: Recent successes, including an RNN winning the M4 competition, renewed interest in neural networks for forecasting despite limited rigorous benchmark evaluation.The paper also notes that datasets and implementations are often not publicly available.
  • Motivation: ETS and ARIMA remain attractive because they combine accuracy with simplicity, robustness, efficiency, and automation for non-expert users.The paper frames these properties as an important usability benchmark for RNN alternatives.
  • Contributions: The study reviews the literature, implements standard RNN architectures with preprocessing, evaluates them on univariate datasets, and compares them with ETS and ARIMA.It targets insights for non-expert practitioners and restricts comparisons to single-seasonality forecasting problems.

2. Background Study

The background introduces univariate forecasting, neural-network foundations, recurrent architectures, and attention-based extensions. It emphasizes that RNN feedback loops model temporal dependencies while reviewing architectures and mechanisms used in forecasting research.

  • Univariate Forecasting: Univariate forecasting predicts future values of one time series from its own past values over a specified forecasting horizon.The formulation represents future values as a model function of the observed series plus approximation error.
  • Traditional Forecasting: Traditional univariate methods such as ETS and ARIMA performed strongly against computational-intelligence methods in several forecasting competitions.The background contrasts these established statistical approaches with newer neural methods.
  • Artificial Neural Networks: Artificial neural networks can model unknown and nonlinear relationships with few a-priori assumptions and can generalize learned relationships to unseen data.They are described as universal approximators with a broader function range than the statistical techniques discussed.
  • Recurrent Neural Networks: Unlike feed-forward networks, RNNs use feedback loops that preserve temporal order and address dependencies across sequence inputs.Feed-forward networks treat each new input window in isolation and carry no state forward.
  • Recurrent Units: Common recurrent units include Elman RNN, LSTM, and GRU cells, while prior empirical comparisons of cells did not specifically cover time-series forecasting.The literature also contains variants such as depth-gated LSTM, Clockwork RNN, stochastic recurrent networks, and bidirectional RNNs.
  • Forecasting Architectures: Forecasting research uses stacked and sequence-to-sequence architectures, including encoder-decoder models and attention mechanisms for selecting relevant inputs or temporal information.Reviewed examples include input, spatial, temporal, and position-based content attention mechanisms.
  • Forecasting Architectures: Ensemble approaches combine diverse base learners through a meta-learner to produce final forecasts.One reviewed stacking method combines multiple LSTM learners and enforces diversity through hyperparameter variation.

3. Methodology

The study implements and compares recurrent units and forecasting architectures, including Elman RNN, peephole LSTM, GRU, stacked networks, and sequence-to-sequence models. The architectures use recurrent state propagation, multiple layers or encoder–decoder components, and output projections suited to forecasting horizons.

  • Methodology: The methodology implements several RNN architectures and recurrent units for a comprehensive experimental comparison.The study includes Elman RNN, LSTM, and GRU units across stacked and sequence-to-sequence architectures.
  • Recurrent Units: The Elman RNN uses feedback from the previous hidden state and current input to update its recurrent state.Its single hidden state is the cell’s only memory, but long sequences can produce vanishing or exploding gradients.
  • Recurrent Units: The peephole LSTM adds an internal cell state and gates that regulate information retention, input, and output.Peephole connections allow input and forget gates to inspect the previous cell state, while the output gate can inspect the current cell state.
  • Recurrent Units: The GRU is a simpler recurrent unit with update and reset gates and a single hidden-state component.Its update gate combines the roles of the LSTM input and forget gates, while the reset gate controls prior hidden-state contribution.
  • Stacked Architecture: The stacked architecture repeats the same recurrent unit across time and can place multiple recurrent layers above one another.A dense affine layer projects recurrent outputs to the forecasting horizon, while higher layers receive outputs from the layer immediately below.
  • Sequence to Sequence Architecture: The sequence-to-sequence architecture uses an encoder to build a state and a decoder whose cells generate forecasts across the horizon.Decoder autoregressive connections feed previous outputs into later steps, and the decoder begins from the encoder’s final context state.

4. Experimental Framework

The experimental framework evaluates standard RNN architectures for univariate, multi-step forecasting across several competition datasets, using preprocessing, validation, automated tuning, and benchmark models. It includes deseasonalization, moving-window inputs, parameter-capacity controls, and adjusted performance measures for series containing zeros.

  • Datasets: The study uses six competition datasets spanning CIF 2016, NN5, M3, M4, Wikipedia Web Traffic, and Tourism forecasting tasks.The experiments select specific categories and series from these datasets for univariate forecasting.
  • Experimental scope: The experiments are restricted to univariate, multi-step forecasting with a single seasonal cycle, using weekly seasonality for daily datasets and yearly seasonality for selected monthly datasets.The NN5 and Wikipedia datasets use weekly seasonality, while selected M3 and M4 monthly series use yearly seasonality.
  • Datasets: The datasets differ substantially in length, with short CIF 2016, M3, and Tourism series, longer NN5 and Wikipedia series, and M4 monthly series ranging from 42 to 2794 observations.These differences provide varied data-length conditions for evaluating the forecasting models.
  • Training and validation: Validation reserves a final segment equal to the forecast horizon, while the remaining observations are used for training and hyperparameter selection.The split uses a fixed-origin mechanism illustrated in Figure 10.
  • Preprocessing: Prior deseasonalization applies variance stabilization followed by STL decomposition, assuming fixed seasonality across each series.STL separates seasonal, trend, and remainder components but is used deterministically here.
  • RNN inputs: The moving-window strategy sets the output window to the forecasting horizon and uses recent inputs to reduce the recurrent unit’s need to retain the full history.The input window is intended to emphasize recent information because distant-step information can fade.

5. Analysis of Results

Across diverse datasets, RNNs were competitive with ETS and auto.arima, but their relative advantage depended on dataset characteristics, architecture, and preprocessing choices.

  • The datasets varied in seasonality strength, series count, and individual-series length, supporting comparisons across diverse time-series characteristics.Results were reported using mean SMAPE and additional error metrics.
  • Pooled regression models with window-sized lags produced mean SMAPE values of 12.89 on CIF, 47.47 on Kaggle, 14.36 on M3, 22.41 on NN5, 21.10 on Tourism, and 13.75 on M4.These values are reported for the Pooled Regression Lags Window model.
  • Unpooled regression results varied substantially, including mean SMAPE values of 15.35, 75.39, 14.81, 30.06, 27.39, and 13.38 for the listed datasets.The values correspond to the Unpooled Regression Lags 10 model row.
  • RNN models were not significantly better than both ETS and auto.arima on CIF2016, M3, Tourism, or M4.On NN5, RNNs significantly outperformed auto.arima but not ets; significant RNN advantages were mostly associated with Stacked variants.
  • Increasing regression lags from 10 to the RNN input-window size generally improved regression accuracy across datasets.Pooled regression also benefited from cross-series information on Wikipedia Web Traffic, M3, and NN5, while RNNs mostly outperformed regression models on CIF, Tourism, and M4.
  • Pooled regression variants with Bayesian or ordinary regularization had nearly identical mean SMAPE results to the unregularized window-lag model.The study reports that selected L2 regularization values were often close to zero, so it could not conclude that regularization improved pooled-model performance.

5.1. Relative Performance of RNN Architectures

Architecture rankings depended on the error metric: S2SD without a moving window performed best on mean SMAPE, whereas Stacked generally performed best on other metrics and most series.

  • S2SD without the moving window performed best on mean SMAPE, while the Stacked architecture performed best on the other error metrics.The violin plots compare relative architecture performance across datasets.
  • The Stacked architecture produced the best results for most time series, although its mean SMAPE was affected by higher errors on some outlying series.The two S2SD variants performed comparably, but generally below Stacked on most cases.
  • Mean SMAPE differences across architectures were statistically significant, with a Friedman-test p-value of 9.3568 × 10^-3.Stacked and S2SD Non MW were not significantly worse than S2SD MW, whereas S2S was significantly worse with adjusted p-value 5.24 × 10^-3.
  • Median SMAPE differences across architectures were not statistically significant, with a Friedman-test p-value of 0.349.

5.2. Performance of Recurrent Units

Recurrent-unit performance differences were not statistically significant on mean SMAPE, although LSTM with peephole connections appeared strongest in the plotted comparisons.

  • LSTM with peephole connections performed best visually on mean SMAPE ranks, while ERNN performed worst and GRU was intermediate.The Friedman test for mean SMAPE produced an overall p-value of 0.101, which does not indicate statistical significance.

5.3. Performance of Optimizers

COCOB was preferred among the tested optimizers because it performed best in the study and does not require an initial learning-rate setting, despite weak significance evidence.

  • COCOB performed best among the three optimizers, while Adagrad performed worst in the plotted comparisons.Adam had been the best optimizer in earlier results, but COCOB was preferred here.
  • The Friedman test found no strong optimizer significance evidence for mean SMAPE, with an overall p-value of 0.115.COCOB was nevertheless preferred because it eliminates the need to tune an initial learning rate.

5.4. Performance of the Output Components for the Sequence to Sequence Architecture

For the sequence-to-sequence architecture, the dense output layer outperforms the decoder, whose teacher-forced autoregressive connections accumulate forecast errors.

  • The dense layer performs better than the decoder in the sequence-to-sequence architecture.Teacher forcing in the decoder carries errors forward across prediction steps, increasing forecast uncertainty.
  • 2.064 × 10^-4 overall p-value indicates that the decoder performs significantly worse than the dense layer on mean SMAPE.

5.5. Comparison of Input Window Sizes for the Stacked Architecture

For the Stacked architecture, large input windows generally improve forecasting, especially when seasonality is not removed, whereas small windows can be comparable under mean MASE.

  • Large input windows improve the Stacked architecture with and without STL Decomposition on the NN5 and Wikipedia Web Traffic datasets.Large denotes a window slightly larger than the prediction horizon; Small denotes one slightly larger than the seven-step daily seasonality period.
  • 2.206 × 10^-3 overall p-value indicates that small input windows perform significantly worse than large windows on mean SMAPE.
  • When seasonality is not removed, large input windows improve accuracy by a huge margin by helping the architecture learn seasonal patterns.

5.6. Analysis of Seasonality Modelling

Seasonality preprocessing is dataset-dependent: STL Decomposition works better for datasets with heterogeneous seasonal patterns, while RNNs can model homogeneous seasonality directly.

  • Wikipedia Web Traffic shows nearly equal performance with and without seasonality removal because its series have minimal seasonality.
  • NN5 has strong, consistent seasonality across similarly aligned series, supporting direct seasonality modelling by NNs.
  • RNNs model seasonality effectively when dataset series have similar seasonal patterns, equal lengths, and coinciding start and end dates.
  • STL Decomposition ranks best on the CIF, M3, and Tourism datasets, whereas omitting it ranks best on Wikipedia Web Traffic and NN5.These comparisons use mean SMAPE across available models and paired Wilcoxon signed-rank tests.

5.7. Performance of RNN Models Vs. Traditional Univariate Benchmarks

RNN performance relative to ETS and ARIMA varies by metric and dataset: RNNs are competitive broadly, but ARIMA remains strongest on the M4 monthly dataset.

  • RNNs outperform traditional benchmarks on all datasets except M4 monthly for both SMAPE and MASE.
  • On M4 monthly, some RNNs outperform ETS, but ARIMA performs better than all RNNs; RNNs lead only in the Micro category.
  • Mean error metrics can be dominated by outlier errors, as shown by RNNs beating ETS and ARIMA on Tourism median SMAPE but not mean SMAPE.
  • The Stacked LSTM COCOB combination outperforms both univariate benchmarks on CIF 2016, NN5, and Wikipedia Web Traffic.

5.8. Experiments Involving the Total Number of Trainable Parameters

The study compares recurrent cell types under matched trainable-parameter budgets and finds results consistent with comparisons based on cell dimension. LSTM cells with peephole connections perform best, while ERNN performs worst and GRU is intermediate.

  • Recurrent-unit comparison: LSTM cells with peephole connections perform best, ERNN performs worst, and GRU performs between them under matched parameter counts.The comparison uses mean SMAPE and mean MASE ranks.
  • Recurrent-unit comparison: Figure 23 compares the relative performance of recurrent cell types under the same number of total trainable parameters.The figure reports comparisons using mean SMAPE and mean MASE ranks.
  • Comparison with cell-dimension tuning: The parameter-tuned results are coherent with results obtained by tuning cell dimension instead.The paper notes a direct connection between trainable-parameter count and cell dimension.

5.9. Comparison of the Computational Costs of the RNN Models Vs. the Benchmarks

RNN forecasting pipelines require substantially more computation than ETS and auto.arima, chiefly because of automated hyperparameter tuning. However, final-model training and testing can be comparable to benchmark costs, while the RNNs outperform benchmarks in the reported cases.

  • Experimental scope: The computational comparison covers RNN models and benchmarks on CIF2016, Wikipedia Web Traffic, M3, and NN5 datasets where an RNN outperformed the benchmarks.Table 8 reports stage-level and total computational times in seconds.
  • Overall computational cost: RNN pipelines took substantially longer overall than the standard benchmarks, including 3749.5 s versus 85.1 s for auto.arima and 41.6 s for ets on CIF horizon 12.On NN5, the RNN required 51490.7 s, compared with 1067.7 s for auto.arima and 53.3 s for ets.
  • Computational-cost breakdown: Most RNN computational time was devoted to SMAC hyperparameter tuning with 50 iterations.The pipeline separately measures preprocessing, tuning, and final model training with testing.
  • Accuracy–cost trade-off: RNNs are typically more computationally expensive than traditional univariate techniques, despite outperforming them in all reported Table 8 cases.The paper frames the accuracy gains as potentially beneficial when computational resources are available.

6. Conclusions

The paper develops systematic guidance for applying RNNs to univariate forecasting, including preprocessing, architecture, recurrent-unit, and optimizer choices. Its experiments identify a generally competitive configuration, while the released framework moves RNN fitting toward greater automation.

  • Study contribution: The study systematically evaluates RNN forecasting choices across datasets with diverse characteristics to derive preprocessing and modeling best practices.The evaluated choices include hyperparameters, architectures, recurrent units, and optimizers.
  • Recommended configuration: A Stacked architecture with peephole LSTM cells and the COCOB optimizer, using deseasonalized moving-window data, is generally competitive across many datasets.This is the configuration identified by the experiments.
  • Seasonality handling: RNNs can capture seasonality directly when series have homogeneous seasonal patterns, sufficient lengths, and the same temporal coverage; otherwise deseasonalization should be used.The conclusion distinguishes homogeneous from non-homogeneous seasonal settings.
  • Automation: The procedure is semi-automatic, but fitting RNNs remains less straightforward and automatic than fitting ets and auto.arima.The initial SMAC ranges are approximately shared across datasets, except for minibatch size.
  • Practical conclusion: The paper concludes that RNNs are a good forecasting option capable of producing reliable forecasts that can outperform the benchmarks.The conclusion is supported alongside the released code framework.

7. Future Directions

The paper identifies scope boundaries and directions for extending RNN forecasting beyond its current setting. Future work includes probabilistic, multivariate, multiple-seasonality, and globally-and-locally parameterized models.

  • Scope limitations: The reported results are limited to point forecasts in a univariate context.The paper notes that probabilistic forecasting and multivariate forecasting address broader settings.
  • Seasonality extensions: Higher-frequency data with sufficient length may benefit from modeling multiple seasonalities in a big-data context.The current study considers only single-seasonality forecasting.
  • Global-versus-local modeling: Global neural-network models often suffer from outlier errors for certain series because average global weights may not suit individual requirements.The paper proposes models combining global and local parameters, potentially with ensembling.
  • Architecture selection: Future forecasting applications must clarify when rapidly emerging architectures, including CNNs, are useful and how difficult they are to adapt.The paper highlights this practical uncertainty for forecasting practitioners.
Loading 1909.00590v5…