Recurrent Neural Networks for Time Series Forecasting: Current Status and Future Directions

• Past neural-network forecasting was often dismissed because complex architectures lacked convincing comparisons with simpler statistical methods and performed poorly in several competitions.Short or unstable time series and neural networks’ black-box nature further limited their practical use.Source: §1 · p. 1
• Many related time series can create a Big Data setting where isolated univariate models are inadequate, while neural networks can learn from massive collections of similar series.Source: §1 · pp. 1–2
• RNNs gained renewed credibility after winning the M4 competition (Smyl, 2020), but existing neural-network methods were often insufficiently benchmarked, unreproducible, and difficult to adapt.Source: §1 · p. 2
• Practitioners still need automated, competitive, and accessible alternatives because no established guidance identifies when RNNs outperform statistical methods or how to select and tune architectures.ETS and ARIMA remain popular because they are accurate, simple, robust, efficient, and automatic for non-expert users.Source: §1 · p. 2
• The study contributes a literature review, rigorous univariate evaluations of standard RNN architectures against ETS and ARIMA, generic tuning guidelines, and a cohesive open-source software framework.The experiments focus on single-seasonality problems and publicly available datasets, excluding specialized competition methods that are not straightforwardly automated.Source: §1 · pp. 2–3

2.1. Univariate Forecasting

• Univariate forecasting predicts a single series from its own history, mapping observations through a model function to a future horizon while accounting for approximation error.The target is {x1, x2, ..., xT}, and forecasts cover steps T + 1 through T + H.Source: §2.1 · p. 3

2.2. Traditional Univariate Forecasting Techniques

• Traditional methods such as ETS and ARIMA are accurate with limited data and few parameters, but per-series models require costly retraining and cannot exploit cross-series information.These limitations matter when individual series are short, whereas learning across many series can address limited data availability.Source: §2.2 · p. 4

2.3. Artificial Neural Networks

• ANNs offer flexible nonlinear function approximation and generalization, but their forecasting accuracy depends critically on network structure, training procedures, and input-variable selection.FFNNs process independent input windows without carrying state, motivating RNNs for time-dependent data; hybrid ARIMA–ANN models separately capture linear and nonlinear components.Source: §2.3 · pp. 4–5

2.3.1. Recurrent Neural Networks for Forecasting

• RNNs preserve temporal order and dependencies through feedback loops, with Elman, LSTM, and GRU cells most commonly used for sequence forecasting.LSTM and GRU often perform similarly, while Elman RNNs can train faster; gradient-based recurrent units remain relatively slow because of backpropagation through time.Source: §2.3.1 · p. 5
• Attention and advanced sequence-to-sequence architectures target information loss and seasonality, with position-based attention surpassing alternatives on five of six datasets and GeoMAN outperforming several baselines.Dual-stage attention also outperformed competing models on both tested datasets and was comparatively robust to noise, while GeoMAN adds separate local and global spatial attention for geo-sensory data.Source: §2.3.1 · pp. 7–8

2.4. Leveraging Cross-Series Information

• Global models estimate shared parameters across many series while forecasting each series independently, enabling cross-series learning for related or data-sparse series.Deep forecasting examples include Smyl (2020), Bandara et al. (2020), and Salinas et al. (2019), which combine global weights with local states or parameters.Source: §2.4 · p. 9

3. Methodology

• The study implements recurrent neural networks using three recurrent units: Elman RNN, peephole LSTM, and GRU.These units constitute the recurrent layers evaluated in the experiments.Source: §3 · p. 9, §3.1 · p. 9, §3.1.1 · p. 9

3.1.1. Recurrent Units

• Elman RNN cells use recurrent feedback from previous hidden states but cannot reliably carry long-term dependencies because gradients may vanish or explode over long sequences.The hidden state is the cell’s only memory, with sigmoid hidden-state and tanh output activations.Source: §3.1.1 · p. 10
• LSTM cells address these gradient problems through separate hidden and cell states plus input, forget, and output gates; this study uses peephole connections.Peephole connections allow gates to inspect previous or current cell states during updates.Source: §3.1.1 · pp. 10–12
• GRUs provide a simpler, faster recurrent alternative using only update and reset gates and a single hidden-state component.The update gate combines the roles of the LSTM’s forget and input gates, reducing the number of parameters.Source: §3.1.1 · pp. 12–13

3.1.2.1 Stacked Architecture

• The study tests stacked RNN architectures that share recurrent weights across time, stack multiple layers, and use dense projections to produce the forecasting horizon.Training accumulates time-step errors and applies Backpropagation Through Time once over each sequence.Source: §3.1.2 · p. 13, §3.1.2.1 · pp. 13–15

3.1.2.2 Sequence to Sequence Architecture

• Sequence-to-sequence models encode inputs over time and decode the forecast horizon autoregressively, using teacher forcing during training and generated forecasts during testing.The decoder can accept future exogenous inputs, while a global dense layer is also tested to avoid decoder error propagation.Source: §3.1.2.2 · pp. 15–16

3.2. Learning Algorithms

• The framework compares Adam, Adagrad, and COCOB, whose self-tuned learning rates reduce the need to specify initial learning-rate ranges.Adagrad can shrink learning rates over time, whereas COCOB adapts parameter-specific step sizes through a coin-betting scheme.Source: §3.2 · pp. 17–18

4.1. Datasets

• Experiments use competition datasets spanning CIF 2016, NN5, M3, M4, Wikipedia Web Traffic, and Tourism, with selected subsets restricted to univariate single-seasonality forecasting.NN5 and Wikipedia use weekly seasonality; M3 and M4 use monthly series with yearly seasonality, while Tourism uses monthly series.Source: §4.1 · p. 18
• The datasets differ substantially in scale, horizon, and series length, including 48,000 monthly M4 series, 1,428 monthly M3 series, 997 Wikipedia articles, and 366 monthly Tourism series.The selected datasets include horizons from 6 to 59 observations, and M4 monthly series range from 42 to 2,794 observations.Source: §4.1 · pp. 19–20

4.2.1. Dataset Split

• Validation reserves the final forecast-horizon observations, whereas testing retrains each global model on the complete series; fixed-origin evaluation reflects competition practice and limits computational cost.One model is trained across all available series, although this approach is expected to work best for related or homogeneous series.Source: §4.2.1 · pp. 20–21, §4.3 · p. 25

4.2.2. Addressing Missing Values

• Missing values are replaced with weekday-specific medians for NN5 and zeros for Wikipedia traffic, which contains more missing values and does not distinguish missing observations from zeros.The NN5 choice is motivated by its strong weekly seasonality.Source: §4.2.2 · p. 21

4.2.3. Modelling Seasonality

• Seasonality experiments compare raw and deseasonalized inputs using deterministic STL decomposition, removing fixed seasonal components while leaving stochastic seasonality for the RNN.STL requires at least two full periods; shorter series are treated as nonseasonal, and multiplicative components are first converted through variance stabilization.Source: §4.2.3 · p. 22

4.2.4. Stabilizing the Variance in the Data

• Because STL requires additive components and logarithms cannot process zero values directly, the experiments use a Log transformation for their non-negative data.Box-Cox and Yeo-Johnson alternatives are described, but their parameter-selection complexities motivate using the Log transformation.Source: §4.2.4 · pp. 22–23

4.2.5. Multiple Output Strategy

• All RNN architectures use multiple-output forecasting, with output windows equal to the horizon and input windows set to 1.25 times either the horizon or seasonality period.Moving windows reduce the recurrent unit’s need to retain the full history and help model remaining stochastic seasonality.Source: §4.2.5 · pp. 23–24

4.2.6. Trend Normalization

• Trend normalization subtracts the final input-window trend for moving-window models, while non-moving-window models use the final training-sequence trend; non-STL experiments instead divide by the series mean before logging.Training tunes minibatch size, epochs, learning rate, noise, L2 regularization, RNN dimensions, hidden layers, and initializer variance.Source: §4.2.6 · p. 24, §4.2.7 · pp. 24–25, §4.3.1 · p. 26

4.3.1. Hyper-parameter Tuning

• Hyperparameter tuning compares grid and random search while primarily optimizing architecture through cell dimension, with an additional parameter-count-based fairness experiment across recurrent units.The latter uses the best cell-dimension model combination and equal initial parameter ranges across LSTM, ERNN, and GRU models.Source: §4.3.1 · p. 26

4.3.1.1 Bayesian Optimization

• Bayesian optimization reduces expensive validation-error evaluations by selecting new hyperparameter configurations through an acquisition function informed by prior evaluations.The approach models the objective function probabilistically and uses Expected Improvement in the cited implementation.Source: §4.3.1.1 · p. 27

4.3.1.2 Sequential Model based Algorithm Configuration (SMAC)

• SMAC implements sequential model-based optimization with a Tree-structured Parzen Estimator, supporting candidate sets, conditional and categorical hyperparameters, time or iteration bounds, and version 0.8.0.The trainable-parameter experiment uses an initial range of 2000–25000 across compared models and datasets.Source: §4.3.1.2 · p. 27

4.3.2. Dealing with Model Overfitting

• To limit overfitting, the framework adds zero-mean Gaussian input noise and L2 weight regularization, with the noise standard deviation and regularization strength tuned as hyperparameters.The L2 penalty discourages excessive weight growth; too much regularization can underfit, while too little can overfit.Source: §4.3.2 · pp. 27–28

4.4. Model Testing

• After tuning, models are retrained on all available data; ten TensorFlow graph seeds generate forecasts that are combined by taking their median.Hyperparameters are held fixed across seeds because repeating tuning would be computationally expensive.Source: §4.4 · p. 28

4.4.1. Data Post-Processing

• Forecast post-processing reverses preprocessing transformations, including normalization, deseasonalization, logarithms, scaling, rounding, and nonnegativity clipping as applicable.STL-based forecasts restore trend and seasonality, whereas non-STL forecasts reverse mean scaling instead.Source: §4.4.1 · pp. 28–29

4.4.2. Performance Measures

• Evaluation uses mean, median, and rank versions of SMAPE and MASE, replacing SMAPE’s unstable near-zero denominator with a variant using ϵ = 0.1 when needed.MASE directly compares performance with a naïve benchmark, while dataset-level ranks enable comparisons across differing metric scales.Source: §4.4.2 · p. 29

4.5. Benchmarks

• Benchmarks include default ets and auto.arima, pooled and unpooled regression models, and regression variants matched to RNN lag structure and L2-regularization procedures.Pooled regression isolates gains from global cross-series modeling, while recursive forecasting produces each horizon one step at a time.Source: §4.5 · p. 30

Experimental scope

• The experiments span datasets differing in seasonality strength, number of time series, and individual-series length, with results reported primarily using mean SMAPE.Other error metrics are provided in the Online Appendix, and Table 5 uses abbreviations for decomposition, windowing, and input-window variants.Source: §5 · p. 31

Benchmark comparison

• Table 5 shows no model significantly beats both benchmarks on CIF2016, M3, Tourism, or M4; NN5 RNNs beat auto.arima but not ets, mainly with Stacked architectures.Only pooled regression models significantly outperform auto.arima on Wikipedia Web Traffic.Source: Table 5 · p. 33, §5 · p. 33

Regression and cross-series effects

• Increasing regression lags to the RNN input-window size generally improves accuracy, while pooled cross-series information helps on Wikipedia Web Traffic, M3, and NN5.On Wikipedia Web Traffic and NN5, pooled increased-lag regression also outperforms auto.arima.Source: §5 · p. 33

RNN architecture effects

• RNNs mostly outperform both regression variants on CIF, Tourism, and M4, and outperform statistical benchmarks and regressions on Wikipedia Web Traffic, M3, and NN5.On CIF, the choice of RNN architecture alone surpasses ets and auto.arima despite limited series homogeneity for global models.Source: §5 · p. 33

Regularization analysis

• Pooled regression regularization does not consistently improve performance because tuning often selects L2 values near zero, whereas unpooled models show mixed results.The unregularized pooled model is already an appropriate fit, while unpooled regularization varies across series.Source: §5 · p. 33

• S2SD without the moving window performs best on mean SMAPE, while Stacked performs best on all other error metrics.Source: §5.1 · p. 33
• Stacked produces the best results for most time series, whereas the two S2SD variants perform comparably and S2S performs worst overall.Source: §5.1 · p. 34
• Mean-SMAPE differences are statistically significant overall (p = 9.3568 × 10−3), but Stacked and S2SD without the moving window are not significantly worse than S2SD with the moving window (adjusted p = 0.602).Source: §5.1 · pp. 34–35
• S2S performs significantly worse than the control architecture on mean SMAPE (adjusted p = 5.24 × 10−3), while median-SMAPE differences are not statistically significant (p = 0.349).Source: §5.1 · p. 35

• The Friedman test on mean SMAPE values yields an overall p-value of 0.101, providing no statistically significant difference.Source: §5.2 · p. 35
• LSTM with peephole connections performs best, GRU is intermediate, and ERNN performs worst across the recurrent-cell comparison.The comparison uses mean SMAPE and mean MASE ranks shown in Figure 13.Source: Figure 13 · p. 35, §5.2 · p. 35
• Figure 13 compares the recurrent units using violin plots of mean SMAPE ranks and mean MASE ranks.Source: Figure 13 · p. 35, §5.2 · p. 35

5.3. Performance of Optimizers

• COCOB performs best among the three optimizers, with Adam competitive and Adagrad worst across mean SMAPE and MASE ranks.Figure 14 compares the optimizers using both mean SMAPE ranks and mean MASE ranks.Source: §5.3 · p. 35
• The Friedman test found no strong significance for mean SMAPE values (p = 0.115), but COCOB is preferred because it requires no initial learning rate.This removes one external hyperparameter and supports fully automating the forecasting approach.Source: §5.3 · p. 35

Output components

• The dense layer performs better because teacher forcing in the decoder propagates errors across forecasting steps, increasing uncertainty along the prediction horizon.The decoder’s autoregressive connections carry forward errors generated at each forecasting step.Source: §5.4 · pp. 35–36
• 2.064 × 10−4 overall p-value indicates that the decoder performs significantly worse than the dense layer under the paired Wilcoxon signed-rank test on mean SMAPE.The comparison is shown in Figure 15.Source: Figure 15 · p. 36, §5.4 · p. 36

5.5. Comparison of Input Window Sizes for the Stacked Architecture

• Mean SMAPE ranks show small windows perform significantly worse overall than large windows, with a paired Wilcoxon test p-value of 2.206 × 10−3.The result combines cases with and without STL Decomposition; large windows also make it easier to learn underlying seasonal patterns, while small windows remain comparable by mean MASE ranks.Source: §5.5 · pp. 36–37
• Large input windows improve the Stacked architecture’s accuracy on NN5 and Wikipedia Web Traffic with and without STL Decomposition.Figure 16 compares windows where “Large” is slightly longer than the prediction horizon and “Small” is slightly longer than the seven-day seasonality period.Source: §5.5 · p. 36, Figure 16 · p. 37

• The comparison evaluates models with and without STL Decomposition using mean SMAPE and mean MASE.Results are presented in Figures 17 and 18, respectively, while the large M4 monthly dataset uses only selected models without prior seasonality removal and reports the comparison in Table 5.Source: §5.6 · p. 37, Figure 17 · p. 38, Figure 18 · p. 39
• On Wikipedia Web Traffic, neural-network seasonality modelling is almost as effective as prior removal except for a few outliers, consistent with minimal seasonality.On NN5, neural networks also reasonably model seasonality independently, apart from a few outliers.Source: §5.6 · p. 40
• Neural networks model seasonality effectively when series share similar seasonal patterns, equal lengths, and coinciding start and end dates.NN5 exemplifies this aligned structure, with nearly identical seasonal patterns and dates across series, whereas Tourism and other datasets show greater variation.Source: §5.6 · p. 40
• STL Decomposition ranks best on CIF, M3, and Tourism, whereas omitting it ranks best on Wikipedia Web Traffic and NN5.The paired Wilcoxon tests find STL significantly better on CIF, M3, and Tourism, while STL is significantly worse on Wikipedia Web Traffic and neither approach differs significantly on NN5.Source: §5.6 · p. 37, Table 6 · p. 40

5.7. Performance of RNN Models Vs. Traditional Univariate Benchmarks

• RNNs outperform traditional univariate benchmarks on all datasets except M4 monthly for both SMAPE and MASE; on M4 monthly, ARIMA outperforms every RNN.Within M4 monthly categories, RNNs outperform benchmarks only in the Micro category.Source: §5.7 · p. 42
• Mean error metrics can be dominated by outliers: on Tourism, RNNs beat ETS and ARIMA on median SMAPE but not mean SMAPE.Source: §5.7 · p. 42
• The Stacked LSTM COCOB combination, using peephole cells and the COCOB optimizer, performs competitively on most datasets and beats both benchmarks on CIF 2016, NN5, and Wikipedia Web Traffic.It is plotted with small and large input windows as the representative suggested model combination; dataset-specific best RNNs are also shown when different.Source: §5.7 · p. 42

Experimental setup

• The experiments selected Stacked architectures with COCOB and evaluated the three recurrent units on datasets where RNNs had outperformed statistical benchmarks.For NN5 and Wikipedia Web Traffic, the study used the Stacked architecture without STL decomposition and with an increased input window size; results were reported using mean SMAPE.Source: §5.8 · p. 42

Results

• Peephole LSTM performed best, ERNN worst, and GRU intermediate under equal trainable-parameter budgets, but the difference was not statistically significant (Friedman p=0.174).The comparison used mean SMAPE and mean MASE ranks across CIF, Wikipedia Web Traffic, M3, and NN5 experiments.Source: §5.8 · p. 42, Figure 23 · p. 46

Interpretation

• Results from tuning total trainable parameters were coherent with tuning cell dimension, because parameter count is directly connected to cell dimension and cell dimension is not very sensitive.Following Smyl (2020), changes in cell dimension and therefore trainable-parameter count do not significantly affect RNN performance.Source: §5.8 · p. 46

5.9. Comparison of the Computational Costs of the RNN Models Vs. the Benchmarks

• The computational-cost comparison decomposes the RNN pipeline into preprocessing, hyperparameter tuning, and final training with the optimal configuration, alongside total runtime.The comparison covers CIF2016, Wikipedia Web Traffic, M3, and NN5 where an RNN outperforms the benchmarks.Source: §5.9 · p. 46
• RNNs required substantially more total computation than auto.arima and ets: 3749.5 s versus 85.1 s and 41.6 s on CIF horizon 12, and 51490.7 s versus 1067.7 s and 53.3 s on NN5.Source: §5.9 · p. 46
• Although the overall RNN process is computationally costly, final-model training and testing took 565.6 s, compared with 849.8 s for auto.arima and 73.0 s for ets on the same dataset.Computational times used 25 CPU cores for both benchmarks and RNN models, and the RNN pipeline included preprocessing, hyperparameter tuning, and final model training.Source: §5.9 · pp. 46–47
• Across the cases in Table 8, RNNs were computationally more expensive but outperformed the traditional univariate benchmarks, with their accuracy gains presented as beneficial when computational resources are available.The final global RNN models also use fewer parameters and are less complex than building one univariate model per series.Source: §5.9 · p. 47, Table 8 · p. 47

5.10. Hyperparameter Configurations

• The experiments use 50 SMAC iterations and broadly consistent hyperparameter ranges across datasets, while scaling the minibatch-size upper bound according to dataset size.The minibatch lower bound is approximately one-tenth of dataset size; larger datasets receive upper bounds differing by orders of 100s rather than 10s.Source: §5.10 · pp. 47–48
• RNNs use 1–2 hidden layers, reflecting observed strong performance with shallow networks and avoiding overfitting and increased computational complexity.The study attributes deeper networks’ drawbacks to additional parameters and their direct effect on computational cost.Source: §5.10 · p. 48
• The identified learning-rate ranges are 0.01–0.9 for Adagrad and 0.001–0.1 for Adam, while cell dimension and initializer variation have little effect on performance.Large Gaussian-noise standard deviations or L2 regularization values can cause severe underfitting and eliminate the neural network’s forecasting effect.Source: §5.10 · p. 48

Conclusions

• Global RNNs exploit cross-series information, which benefits some datasets, while their modelling capacity can preserve competitive accuracy even across heterogeneous series.The study evaluates global models because they leverage large databases containing many related time series.Source: §6 · p. 48
• Stacked LSTM models with peephole connections and the COCOB optimizer, fed deseasonalized moving-window data, are generally competitive across datasets and can outperform statistical benchmarks.The LSTM is the best unit type even after correcting for trainable parameters, although the difference is not statistically significant.Source: §6 · p. 48
• RNNs capture seasonality without deseasonalization when all series share homogeneous seasonal patterns and sufficient common duration; otherwise, deseasonalization is recommended.RNNs are weak at modelling seasonality alone when these conditions do not hold.Source: §6 · p. 48
• RNNs require relatively more computation than ETS and auto.arima and remain less straightforward to fit, but cloud infrastructure makes their processing times feasible and the released framework advances automation.The procedure uses approximately common initial SMAC hyperparameter ranges across datasets, varying minibatch size with dataset size.Source: §6 · pp. 48–49

7. Future Directions

• The study’s scope should be broadened to probabilistic, multivariate, and multiple-seasonality forecasting for complex and high-frequency data.These extensions would model prediction uncertainty, interdependent product series, and multiple seasonal patterns.Source: §7 · p. 49
• Global neural-network models can produce outlier errors on individual series, motivating hierarchical models that combine global and local parameters, potentially with ensembling.Bandara et al. (2020), Smyl (2020), and Sen et al. (2019) are identified as steps toward this largely open research problem.Source: §7 · p. 49
• Future research should clarify when CNN-based architectures are preferable to RNNs, as CNN filters, dilated convolutions, residual connections, and TCNs target local or long-range dependencies and may train efficiently.Shih et al. (2019), Lai et al. (2018), van den Oord et al. (2016), Borovykh et al. (2018), and Bai et al. (2018) are cited in this line of work.Source: §7 · p. 49