A review and comparison of strategies for multi-step ahead time series forecasting based on the NN5 forecasting competition

Souhaib Ben Taieb; Gianluca Bontempi; Amir Atiya; Antti Sorjamaa

Source-linked AI summary

A review and comparison of strategies for multi-step ahead time series forecasting based on the NN5 forecasting competition

Souhaib Ben Taieb, Gianluca Bontempi, Amir Atiya, Antti Sorjamaa

arXiv:1108.3259v1Aug 16, 2011stat.MLcs.AIcs.LGstat.AP

TL;DR

Multi-step-ahead forecasting is difficult because uncertainty increases with the forecast horizon, while existing models have received limited attention. This paper reviews and compares forecasting strategies, finding that Multiple-Output strategies perform best and that deseasonalization consistently improves results.

Problem
Multi-step-ahead forecasting is difficult because uncertainty increases with the forecast horizon, and existing models have received limited attention.
Method
The paper presents a comparative review of strategies for multi-step-ahead forecasting in theoretical and practical terms.
Results
Multiple-Output strategies are invariably the best strategies, while deseasonalization leads to consistently better results in 38 out of 39 models.
Takeaways & Limitations
The overall best method is MIMO-ACFLIN used with input selection and deseasonalization.
Takeaways & Limitations
Configurations requiring input selection do not report results for DIRMO because input selection was not combined with parameter-s selection.

Abstract

from arXiv · show

Multi-step ahead forecasting is still an open challenge in time series forecasting. Several approaches that deal with this complex problem have been proposed in the literature but an extensive comparison on a large number of tasks is still missing. This paper aims to fill this gap by reviewing existing strategies for multi-step ahead forecasting and comparing them in theoretical and practical terms. To attain such an objective, we performed a large scale comparison of these different strategies using a large experimental benchmark (namely the 111 series from the NN5 forecasting competition). In addition, we considered the effects of deseasonalization, input variable selection, and forecast combination on these strategies and on multi-step ahead forecasting at large. The following three findings appear to be consistently supported by the experimental results: Multiple-Output strategies are the best performing approaches, deseasonalization leads to uniformly improved forecast accuracy, and input selection is more effective when performed in conjunction with deseasonalization.

1. Introduction

The introduction frames multi-step ahead forecasting as a difficult, underexamined problem and reviews five strategies proposed for generating forecasts. The paper unifies these strategies theoretically and compares their performance experimentally on the 111 NN5 series.

Multi-step ahead forecasting is harder than one-step forecasting because longer horizons increase uncertainty and can cause error accumulation, reduced accuracy, and greater uncertainty.
Existing machine-learning approaches for generating multi-step forecasts received limited attention, leaving conflicting evidence about which strategy performs best.
Forecasting strategies: The Recursive strategy repeatedly applies one-step forecasts, feeding each predicted value back as input for the next forecast.
Forecasting strategies: The Direct strategy fits H separate models, with each model forecasting one specific future position.
Forecasting strategies: DirRec combines Direct and Recursive strategies by using a different model at each step while adding previous approximations to later inputs.
Forecasting strategies: MIMO predicts a vector of future values jointly to preserve stochastic dependency between forecasts, while DIRMO seeks a trade-off between this dependency preservation and modeling flexibility.
Paper contributions: The paper presents a unified theoretical review and compares forecasting strategies experimentally on the 111 time series from the NN5 international forecasting competition benchmark.

2. Strategies for Multi-Step-Ahead Time Series Forecasting

Multi-step-ahead forecasting predicts the next H observations and can use recursive, direct, hybrid, or multiple-output strategies. These strategies differ in how many models they learn, how forecasts depend on one another, and how they trade error accumulation against dependence modeling and flexibility.

A multi-step-ahead task predicts the next H values of a historical series, where H > 1 is the forecasting horizon.
Recursive strategy: Recursive forecasting trains one one-step model and repeatedly feeds forecasts back as inputs for later horizons.This can accumulate errors, especially when h exceeds the embedding dimension d and later inputs are all forecasted values.
Direct strategy: Direct forecasting learns H independent models, avoiding accumulated forecast errors but neglecting dependencies among horizon-specific forecasts.Its independent models can also produce incoherent trajectories and require as many models as the horizon length.
DirRec strategy: DirRec combines direct and recursive designs by learning a different model for each horizon while adding previous forecasts to later inputs.Its embedding size can vary across horizons, and prior work reported better performance than Direct and Recursive strategies on two real-world series.
MIMO strategy: MIMO learns one multiple-output model that forecasts the horizon jointly, preserving stochastic dependencies and avoiding Direct independence and Recursive error accumulation.Its drawback is that every horizon shares the same model structure, potentially reducing flexibility.
DIRMO strategy: DIRMO forecasts the horizon in blocks, decomposing the task into n = H/s multiple-output tasks and interpolating between Direct and MIMO.s = 1 corresponds to Direct, whereas s = H corresponds to MIMO, with intermediate values yielding intermediate configurations.

3. Lazy Learning for Time Series Forecasting

The paper compares forecasting strategies using a common learning algorithm and focuses on Lazy Learning as a local, query-based model. Lazy Learning defers fitting until prediction, uses nearby temporal patterns, and offers reduced assumptions, online learning, and nonstationarity modeling.

The experiments compare multi-step forecasting strategies rather than forecasting models, so the paper adopts Lazy Learning as the underlying algorithm.Lazy Learning is selected as a local model because prior work found it effective for time-series forecasting.
Local models approximate the input-output function near the query point using nearby data, unlike global models that describe the whole input domain analytically.Examples include nearest-neighbor, weighted-average, and locally weighted-regression models.
Lazy Learning defers the learning procedure until a forecast is requested, then searches for neighbors, estimates a local model, and discards it afterward.The process repeats from scratch for subsequent queries, making the procedure query-based and local.
Lazy Learning assumes no a priori knowledge of the underlying process and provides reduced assumptions, online learning capability, and capacity to model nonstationarity.These properties motivate its use in a multi-step-ahead forecasting context.
The local-learning setup uses k as the critical selectable parameter because it controls the bias/variance trade-off.The implementation fixes the tricubic kernel, constant parametric family, and Euclidean distance while limiting automatic selection to the number of neighbors.
For time-series forecasting, the series is embedded as input-output pairs whose inputs are temporal patterns of length d.The output is either a future scalar value for Single-Output Modeling or a future pattern of length H for Multiple-Output Modeling.

4. Experimental Setup

The experimental setup uses the NN5 competition’s 111 daily ATM cash-withdrawal series to compare five multi-step forecasting strategies across preprocessing configurations. Accuracy is assessed with SMAPE, while the configurations vary preprocessing choices such as deseasonalization, input selection, and model selection.

4.1. Time Series Data: The dataset presents real-world challenges including multiple series, outliers, missing values, and overlapping seasonalities.These characteristics motivate evaluating forecasting strategies under complex multi-step conditions.
4.1. Time Series Data: The NN5 competition contains 111 daily ATM cash-withdrawal series, each with roughly two years of data and a 56-day forecasting horizon.Each series has 735 historical data points and represents an ATM in a UK city.
4.1. Time Series Data: Forecast accuracy is evaluated using SMAPE for each series and averaged across series as mean SMAPE.SMAPE compares predictions with target outputs using a relative error measure.
4. Experimental Setup: The study compares five forecasting strategies using multiple configurations to increase the statistical power of the comparison.The configurations vary preprocessing and model-related design choices, including deseasonalization, input selection, and model selection.
4. Experimental Setup: Twelve configurations result from alternative choices for deseasonalization, input selection, and model selection.The setup considers two deseasonalization alternatives, two input-selection alternatives, and three model-selection alternatives.

Step 1: Gaps removal

Gap removal addresses zero withdrawals and missing observations in the NN5 series before forecasting. The preprocessing replaces gaps using available nearby or yearly seasonal reference values, while deseasonalization removes selected seasonal patterns before modeling and restores them afterward.

Step 1: Gaps removal: Gaps represent either zero withdrawals or missing observations, and they affect about 2.5% of the data.The preprocessing treats both anomaly types as gaps requiring removal.
Step 1: Gaps removal: The adopted gap-removal method replaces each gap with the median of available values at offsets of ±365 or ±7 days.The reference set is [y_m+365, y_m−365, y_m+7, y_m−7].
Step 1: Gaps removal: Deseasonalization is included because NN5 series contain varied periodic patterns that may affect forecasting strategies.The experiments therefore compare tasks with and without deseasonalization.
Step 1: Gaps removal: The preprocessing removes strong day-of-week and moderate day-of-month seasonality before forecasting, then restores the seasonality afterward.The forecasting model is applied to the deseasonalized series, followed by seasonal reconstruction.

Step 3: Embedding dimension selection

Embedding dimension selection determines the lagged input representation for each forecasting strategy. Because it is not central to the paper, the study uses a reviewed PACF-based approach, producing an average final input dimensionality of 24.

Step 3: Embedding dimension selection: Every forecasting strategy requires setting an embedding dimension that determines the lagged input representation.The paper treats this design choice as necessary for all five strategies.
Step 3: Embedding dimension selection: Embedding-dimension selection is not a central theme, so the study applies a state-of-the-art reviewed approach rather than developing or comparing alternatives.The selected approach is based on significant partial correlation functions.
Step 3: Embedding dimension selection: The study selects time-lagged realizations with significant PACF values to determine the embedding dimension and relevant past variables.The maximum PACF lag is set to 200 to provide a broad feature pool.
Step 3: Embedding dimension selection: The final input vectors have an average dimensionality of 24 across all time series.This is the reported average after the lag-selection procedure.

Step 4: Input Selection

Input selection is evaluated as a separate experimental choice using the Delta test and a Forward-Backward Search procedure. The criterion seeks inputs with the strongest deterministic dependence on the forecasting output, while the search explores additions and removals.

Step 4: Input Selection: The experiments compare forecasting tasks with and without input-variable selection.This isolates the effect of selecting a subset of lagged inputs.
Step 4: Input Selection: Input selection uses the Delta test as its relevance criterion and Forward-Backward Search as its search procedure.The selection framework explicitly separates variable-quality assessment from input-space exploration.
Step 4: Input Selection: The Delta test selects inputs associated with the strongest and most deterministic dependence between inputs and outputs.It is based on a noise-variance estimator.
Step 4: Input Selection: Forward-Backward Search combines sequential input addition with sequential removal to explore the input space more flexibly.The procedure is initialized with the variables defined in the preceding step.

Step 5: Model Selection

The experiments compare eight forecasting strategies using three model-selection approaches: choosing the winner, simple averaging, or error-weighted averaging.

Model selection: Three model-selection approaches are considered: WINNER, COMB, and WCOMB.WINNER selects the best test-set model; COMB averages all alternatives; WCOMB uses weights inversely proportional to test errors.
Forecasting strategies: The assessment tests eight forecasting strategies and their variants.The listed strategies include REC, DIR, DIRREC, MIMO variants, and DIRMO-SEL.

7. DIRMO-AVG

The DIRMO-AVG strategy averages forecasts generated with different values of its parameter s, within an evaluation procedure comparing forecasting strategies across 111 series.

7. DIRMO-AVG: DIRMO-AVG calculates a simple average of forecasts obtained with different values of parameter s.
Forecasting performance evaluation: The evaluation procedure is presented as a sequence of steps in Figure 4.
Forecasting performance evaluation: Forecasting-strategy accuracy is measured with SMAPE* across the 111 time series.The evaluation then tests for general performance differences among the strategies using procedures for multiple models and datasets.
Statistical comparison: The comparison uses a two-stage procedure: an omnibus test of equal mean ranks followed by post-hoc pairwise tests when the null hypothesis is rejected.The cited procedure is recommended for comparing multiple models on multiple datasets.

Friedman test

The paper ranks algorithms separately on each dataset and applies Friedman-style tests with post-hoc corrections, using a two-phase NN5 experimental design over 111 time series.

Friedman test: Algorithms receive ranks separately for each dataset, with rank 1 assigned to the best performer and average ranks used for ties.
Friedman test: The Friedman null hypothesis states that all algorithms are equivalent, so their average ranks should be equal.
Post-hoc testing: When significant differences are found, post-hoc tests identify pairwise differences among algorithms.Shaffer’s correction is adopted because it uses logically related hypotheses while retaining the same complexity as Holm’s procedure.
Experimental design: The experiments reproduce NN5 through pre-competition and competition phases.The pre-competition phase compares forecasting strategies, while the competition phase generates final forecasts using the preceding design choices.
Experimental design: Each 735-observation series is partitioned into training, validation, and testing sets of 623, 56, and 56 values.The validation set tunes models, and the test set measures forecasting performance.
Experimental design: Multiple time origins generate forecasts from three starting points, using the same test period and SMAPE criterion as prior work.

5. Results and discussion

The experiments compare forecasting strategies across pre-competition and competition phases using SMAPE*, rankings, and post-hoc significance tests. Multiple-Output strategies, deseasonalization, and input selection emerge as the most consistent performance factors, while computational constraints limit some configurations.

Evaluation design: The evaluation reports average SMAPE* and rankings for 12 configurations, followed by statistical grouping using Friedman and post-hoc tests.The null hypothesis that all algorithms are equivalent was rejected for every configuration.
Best configuration: The overall best method is MIMO-ACFLIN with input selection, deseasonalization, and equal-weight COMB, yielding a competition-phase SMAPE* of 20.28%.The corresponding configuration was considered competitive with the best computational-intelligence competition entries.
Overall findings: Multiple-Output strategies invariably outperform Single-Output strategies, and their relative performance patterns persist from pre-competition to competition results.The persistence also applies to deseasonalization, input selection, and model selection findings.
Strategy comparisons: DIRMO and MIMO achieve comparable performance, while DIRMO-WAVG may be slightly better than the other DIRMO versions.
Strategy comparisons: Among Single-Output strategies, REC generally outperforms DIR, whereas DIRREC performs worst overall, especially without deseasonalization.
Design choices: Deseasonalization improves results in 38 of 39 models, potentially because it reduces the burden of forecasting seasonality alongside trend and other components.
Design choices: Input selection is especially beneficial after deseasonalization, but its effects are mixed without deseasonalization because essential seasonal information may be removed.
Model selection: Combining models is superior to winner-take-all selection, while COMB and WCOMB produce similar results.

6. Conclusion

The paper addresses the difficulty of forecasting farther into the future through a comparative review and extensive NN5 benchmark. Its most consistent findings favor Multiple-Output approaches, deseasonalization, and performance-based strategy selection, while suggesting further work on improved Multiple-Output methods.

Higher forecast horizons bring greater uncertainty, making multi-step ahead forecasting particularly difficult.
The paper combines a comparative review of existing strategies with an extensive comparison on the 111 NN5 forecasting-competition series.
Multiple-Output approaches are invariably better than Single-Output approaches in the reported comparisons.
Deseasonalization has a very considerable positive impact on forecasting performance.
Selecting the best strategy based on testing performance is reported as a potent approach, with future research targeting improved Multiple-Output strategies and tailored deseasonalization.