Recurrent Neural Networks for Time Series Forecasting

• RNNs suit sequential learning because they can use information remembered from the past to predict time-dependent targets.Source: §1 · p. 1
• Despite that fit, applying RNNs to time-series forecasting is not a trivial task.Source: §1 · p. 1
• The article reviews practices that may help RNNs deliver useful forecasts.Source: §1 · p. 1

• Tang et al. [1991] found neural networks outperform Box-Jenkins for short-memory series, while both methods produced similar results for long-memory series.Source: §2 · p. 2
• Faraway and Chatfield [1998] found that architecture and input-variable choices require care, and that increasing hidden nodes may worsen out-of-sample performance.Source: §2 · p. 2
• Zhang and Qi [2005] found neural networks do not capture seasonality by default, while deseasonalization and detrending can help forecasting performance.Source: §2 · p. 2

3 Recurrent neural networks

• RNNs use previous outputs as well as current inputs, while memory-based LSTM and GRU models address the difficulty of training simple recurrent networks.Source: §3 · p. 2

3.1 LSTM

• LSTM uses three gates to manage memory: input and forget gates manage the cell state, and the output gate produces the hidden state.Source: §3.1 · pp. 2–3

3.2 GRU

• GRU is a simpler LSTM variant with two gates: its update gate manages information flow and its reset gate enters the candidate activation.Source: §3.2 · p. 3

3.3 Recurrent neural networks for forecasting

• Prior forecasting work applies LSTMs and GRUs to tasks including anomaly detection, outliers, missing values, and special-event forecasting; this article introduces an end-to-end framework for meaningful forecasts.Source: §3.3 · pp. 3–4

4.1 Feature engineering

• Feature engineering can encode trend, seasonality, deterministic conditions, and target lags; normalization is also described as beneficial for training.Source: §4.1 · p. 4

4.1.1 Lags

• Lagged target values can serve as predictors, but using them loses initial observations and may matter for short series.Source: §4.1.1 · p. 4

4.1.2 Trend

• An equidistant increasing time feature may be enough to represent trend.Source: §4.1.2 · p. 5

4.1.3 Seasonality

• For seasonality, sine and cosine transformations of ordered intervals address cyclicity, and both are needed to represent each interval uniquely.Source: §4.1.3 · p. 5

4.1.4 Dummy indicators

• Binary indicators can encode potentially informative conditions such as holidays, working days, or working hours.Source: §4.1.4 · p. 5

4.2 Feature importances

• Permutation importance measures accuracy drops after variables are shuffled; scores from R2 and directional accuracy are evaluated independently, and repeated encodings can be grouped by their maximum score.Source: §4.2 · pp. 5–6

5.1 Point predictions

• Recursive multi-step forecasting feeds predicted target values back as inputs when lagged targets are the only non-deterministic features, but rolls errors forward.The networks are primarily designed for one-step-ahead forecasting.Source: §5.1 · p. 6

5.2 Prediction intervals

• Bootstrap samples of sequences train separate RNNs, whose predictions are averaged for the final point forecast and used to construct prediction intervals.The approach is similar to Heskes [1997]; bootstrap resampling can also improve forecasts, though bagging may degrade more stable predictors.Source: §5.2 · p. 7
• Confidence intervals quantify uncertainty in the estimated regression mean, whereas prediction intervals target future values and require estimating regression noise variance.The confidence-interval construction assumes the true regression follows a normal distribution given the estimate.Source: §5.2 · pp. 7–8
• A residual-prediction network estimates remaining residuals from inputs, and its predicted variances are added to forecast variances to estimate target-value uncertainty.It is trained on observations left out of the current bootstrap run; an exponential output activation ensures positive predicted error variances.Source: §5.2 · p. 8

6.1 Point forecasts

• About 63.2% of subsequences enter an average bootstrap sample, leaving the rest for validation; a chronological future split supplies the test set.The future test set can also evaluate iterative multi-step forecasts, while both sets assess one-step-ahead forecasts.Source: §6.1 · p. 9
• Forecast fit is assessed with five regression metrics, while directional accuracy uses four classification scores; together, the measures capture different aspects of forecast quality.MAE is easier to interpret than RMSE, MedAE is more robust to outliers, and SMAPE expresses error as a percentage.Source: §6.1 · pp. 9–10

6.2 Interval forecasts

• Interval evaluation combines coverage and width measures, with normalized width supporting comparisons across datasets and lower CWC indicating higher quality.CWC uses nominal confidence level µ and η to magnify the difference between that level and observed coverage.Source: §6.2 · p. 10

7 Empirical study

• The example used two years of hourly Capital Bikeshare rental counts, time-derived features and target lags, while excluding weather information; importances were computed only for one-step forecasts.The cyclical features encoded season, month, week, day, and hour; importances were averaged across four estimates.Source: §7 · p. 10, §7.1 · p. 11

7.1 Features

• Recent target lags were the most important feature group, seasonal variables also mattered, and the LSTM and GRU feature rankings were similar.Trend appeared less important, although it could add forecasting capability during model building.Source: §7.1 · p. 13

7.2 Predictions

• A one-layer, 32-unit LSTM or GRU used 50 bootstrap samples to produce test-set multi-step forecasts with approximate 90% prediction intervals; hyperparameters were not optimized.The models were trained on 70% of the data, with the last 30% held out for testing.Source: §7.2 · p. 13

7.3 Evaluation

• R² was close to 0.95 for single-step and around 0.8 for multi-step forecasts; multi-step RMSE was about twice as large, while directional accuracy was roughly 85% versus 75%.For multiple steps, direction forecasts seemed to work somewhat better than value forecasts.Source: §7.3 · p. 15
• Bagged estimators consistently performed better than individual neural networks on both regression and classification tasks.The final predictions averaged estimates from the bootstrap samples.Source: §7.3 · p. 15
• All prediction intervals exceeded their 90% target coverage; one-step intervals scored better on CWC, while multi-step forecasts achieved similar coverage with wider intervals.The GRU and LSTM had quite similar forecasting performance.Source: §7.3 · p. 17

8 Conclusions and Future Perspectives

• Bagging benefited RNN forecasts, while LSTM and GRU showed about the same forecasting performance.Source: §8 · p. 18
• Permutation importance identified lagged target values as the most important predictors and seasonality features as important, but covered only one-step forecasts.Source: §8 · p. 18
• Bootstrapping produced prediction intervals with consistently higher coverage than targeted, and was computationally expensive.Source: §8 · pp. 18–19
• Multi-step forecasts had higher errors than single-step forecasts, with smaller error gaps for direction-of-change predictions.Source: §8 · p. 18
• Resetting LSTM/GRU cell states at each batch omitted dependencies across batches; the authors expected higher accuracy if those relationships were used.Source: §8 · p. 18
• The authors considered their iterative multi-step forecasting method potentially suboptimal and suggested sequence-to-sequence learning as an alternative.Source: §8 · p. 18
• The authors identify room for additional feature engineering, importance measures across forecast horizons, and deeper study of RNN cell mechanisms.Source: §8 · pp. 18–19
• The study was noncomprehensive, did not optimize hyperparameters, and used a two-year hourly bike dataset described as too small to exploit neural networks; transfer learning is proposed as promising for scarce time-series data.Source: §8 · pp. 18–19