PaperlayerView exact revision on arXiv

Source-linked AI summary

Review of automated time series forecasting pipelines

Stefan Meisenbacher, Marian Turowski, Kaleb Phipps, Martin Rätz, Dirk Müller, Veit Hagenmeyer, Ralf Mikut

arXiv:2202.01712v1cs.LG

TL;DR

Time-series forecasting demand is growing, while tailoring its iterative, expertise-dependent design process is challenging. This review analyzes automation across forecasting-pipeline sections and finds that most of the 63 reviewed papers cover only two or three of five sections, motivating holistic pipeline research.

  • Problem

    Time-series forecasting demand is growing, while tailoring its iterative design process is time-consuming and requires expert knowledge.

    Source: §1 · p. 2

  • Method

    The review analyzes automated forecasting-pipeline literature, including the interaction, combination, and coverage of methods across pipeline sections.

    Source: §9 · p. 23

  • Results

    Most of the 63 reviewed papers cover only two or three of the five forecasting-pipeline sections.

    Source: §9 · p. 23

  • Takeaways & Limitations

    The authors conclude that future research should holistically consider forecasting-pipeline automation to support large-scale application without manual, time-consuming tailoring.

    Source: §9 · p. 23

    Embedded feature selection is designed for its respective training algorithm, so transfer to other forecasting methods is not straightforward.

    Source: §5.4 · p. 11

Abstract

from arXiv · show

Time series forecasting is fundamental for various use cases in different domains such as energy systems and economics. Creating a forecasting model for a specific use case requires an iterative and complex design process. The typical design process includes the five sections (1) data pre-processing, (2) feature engineering, (3) hyperparameter optimization, (4) forecasting method selection, and (5) forecast ensembling, which are commonly organized in a pipeline structure. One promising approach to handle the ever-growing demand for time series forecasts is automating this design process. The present paper, thus, analyzes the existing literature on automated time series forecasting pipelines to investigate how to automate the design process of forecasting models. Thereby, we consider both Automated Machine Learning (AutoML) and automated statistical forecasting methods in a single forecasting pipeline. For this purpose, we firstly present and compare the proposed automation methods for each pipeline section. Secondly, we analyze the automation methods regarding their interaction, combination, and coverage of the five pipeline sections. For both, we discuss the literature, identify problems, give recommendations, and suggest future research. This review reveals that the majority of papers only cover two or three of the five pipeline sections. We conclude that future research has to holistically consider the automation of the forecasting pipeline to enable the large-scale application of time series forecasting.

1 Introduction

Time-series forecasting uses a five-section design pipeline, but tailoring it is iterative, time-consuming, and dependent on expertise. This review compares automation methods across the sections and examines their interactions and coverage.

1 Introduction

  • Forecast design typically covers pre-processing, feature engineering, hyperparameter optimization, forecasting-method selection, and forecast ensembling.Pre-processing transforms data; feature engineering extracts useful characteristics or exogenous information; ensembling bundles forecasts to increase robustness.Source: §1 · p. 2
  • Manual pipeline tailoring is challenging because design choices are iterative, expertise-dependent, and affect forecast accuracy amid growing forecast demand.The paper argues that automation can increase design efficiency.Source: §1 · p. 2
  • A forecasting pipeline organizes automation methods and data flow; running it trains a forecasting method on historical data and returns a parameterized model.The review focuses on automating design decisions, rather than forecast operation such as self-monitoring and automatic model adaptation.Source: §1 · pp. 2–3
  • The review compares automation methods across pipeline sections and analyzes their interactions, combinations, and coverage, including AutoML and automated statistical forecasting.It also identifies problems, gives recommendations, and suggests future research.Source: §1 · p. 3

2 Methodology

The review follows a literature-identification process based on steps proposed by Webster and Watson. It screens relevant time-series forecasting papers for automation of a previously human-intervened pipeline element.

2 Methodology

  • The review methodology adopts fundamental literature-review steps proposed by Webster and Watson.Its stated aim is to identify major contributions, their origin, and evolution.Source: §2 · p. 4
  • The Scopus search yielded 359 hits, and abstract screening retained papers on time-series forecasting that automated at least one previously human-intervened pipeline element.The search considered titles, abstracts, and keywords; listed predatory journals and publishers were excluded.Source: §2 · p. 4
  • Backward-forward citation searching found 2,152 additional papers, reduced to 242 by keyword filtering, and screened those candidates using the same criteria.The search identified papers citing or cited by those that passed screening.Source: §2 · p. 4
  • The review retained 144 papers after abstract screening, 71 after full-text analysis, and 63 after reducing similar work to the broadest-scope articles and proposed improvements.It presents automation methods by pipeline section and discusses gaps and future research directions through pipeline coverage.Source: §2 · p. 4, Discussion · p. 4

3 Time Series Forecasting

The section introduces time-series forecasting models and three method families: naïve, statistical, and machine learning. It also describes strategies for using machine-learning regression to forecast multiple steps ahead.

3 Time Series Forecasting

  • A forecasting model estimates future values over a forecast horizon using current and past time-series values.A time series is a set of observations typically measured at equidistant time points.Source: §3 · p. 4
  • The three method families are naïve, statistical, and machine learning; examples include random walk, ARIMA and exponential smoothing, and SVR, tree-based methods, and neural networks.Statistical methods use historical-data statistics, while machine-learning methods have fewer restrictions regarding linearity and stationarity.Source: §3 · pp. 4–5
  • Machine-learning regression can forecast multiple steps recursively with one model, directly with separate horizon-specific models, or jointly with a multiple-output model.The recursive strategy feeds each output into the next step; the multiple-output strategy returns a vector.Source: §3 · p. 5

4 Data Pre-processing

Pre-processing transforms data before forecasting, including anomaly handling, stationarity testing and transformation, and scaling. The reviewed methods and recommendations vary with the forecasting method and data conditions.

4.1 Anomalies

  • Anomalies include outliers and missing observations, which can degrade forecasting performance or cause training failure; automated methods detect and handle both.Reviewed approaches include neighbor-based thresholds, Hampel filtering, ARIMA residuals, and median- or trend-based missing-value imputation.Source: §4.1 · pp. 5–6

4.2 Stationarity

  • Automated stationarity methods detect trends or seasonality and apply transformations such as differencing, seasonal differencing, frequency filters, or logarithmic transforms.Methods include ACF/PACF decay patterns, unit-root tests, normality tests, and decomposition.Source: §4.2 · pp. 6–7

4.3 Scaling

  • Scaling can help gradient-descent methods converge and prevent high-variance inputs from dominating distance measures; min-max and Z-score methods are outlier-sensitive.Robust scaling instead centers by the median and scales by the interquartile range.Source: §4.3 · pp. 7–8

4.4 Discussion

  • Stationarity transformations are used predominantly with statistical methods, scaling mainly with machine-learning methods, and anomaly handling is relevant to both families.The section’s table captions identify summaries of stationarity and scaling methods; the discussion reports the family-level usage patterns.Source: Table 2 · p. 7, §4.4 · p. 8, Table 3 · p. 8
  • The review recommends handling anomalies before transformations, then testing stationarity when required by the forecasting method, followed by scaling as appropriate.It notes that outlier methods often focus on isolated cases, which may limit performance with consecutive anomalies; future work should consider domain knowledge and test stationarity transformations for machine-learning accuracy.Source: §4.4 · p. 8

5 Feature Engineering

Feature engineering extracts and selects explanatory variables in automated forecasting pipelines. The review finds lag features especially common and recommends broad default feature extraction, automated selection, and domain-specific exogenous features.

5.1 Feature Extraction

  • Feature extraction enriches the feature space with explanatory variables, including prior observations, encoded time cycles, transformations, and domain-dependent exogenous inputs.Examples of transformations include derivatives, decomposition, and moving averages; exogenous inputs may include weather, holidays, or economic indicators.Source: §5.1 · pp. 9–10

5.2 Feature Selection

  • Feature selection removes variables that add no information, using filters, wrappers, embedded methods, or combinations of these approaches.Filters select from data characteristics independently of the forecasting method, whereas wrappers assess candidate features using forecasting performance on validation data.Source: §5.2 · pp. 10–11
  • Wrapper methods commonly require more computation than filters, while empirical studies report better performance for wrappers than filters.Reviewed search approaches include candidate-lag searches, forward selection, optimization, and evolutionary strategies; Donate et al. report that EDA has the best convergence speed and lowest forecast error among the strategies they evaluate.Source: §5.2 · pp. 10–11

5.3 Feature Aggregation

  • Feature aggregation reduces feature-space dimensionality, but PCA-based aggregation assumes normality and may require prior anomaly handling and transformations to obtain stationarity.Aggregation can reduce processing time and help avoid the curse of dimensionality, but it also limits interpretability of feature information for the forecast.Source: §5.3 · p. 11, §5.4 · p. 11

5.4 Discussion

  • Lag features are the most applied features, partly because machine-learning pipelines do not consider time lags implicitly while statistical methods do.The reviewed pipelines use cyclic and transformation features less often, in part because appropriate lags may already represent that information.Source: §5.1 · p. 9, §5.4 · p. 12
  • The review recommends combining filtering with wrapper selection: filtering first reduces candidates, then wrappers tailor selection to the forecasting method or pipeline.Embedded selection is computationally beneficial but is designed for a particular forecasting method, making transfer to other methods not straightforward.Source: §5.4 · pp. 11–12
  • The review does not consider automated feature aggregation necessary for high forecast accuracy because selection can reduce feature count, while aggregation limits interpretability.The authors note that aggregated features often correlate with other features.Source: §5.4 · p. 12
  • The authors recommend extracting a broad set of default endogenous features and extending extraction with domain-specific exogenous features.Automated selection can then remove irrelevant features from the extracted set.Source: §5.4 · p. 12

6 Hyperparameter Optimization

Hyperparameter optimization selects forecasting-model settings by evaluating performance criteria. The section reviews the metrics and validation choices used, then compares several automated search strategies.

6.1 Performance Metrics and Validation Sample

  • Information criteria assess information loss while balancing goodness-of-fit and model complexity; BIC additionally accounts for the number of data points, while MSE and MAE are error measures.Error measures can be evaluated in-sample or out-of-sample; cross-validation averages validation errors across multiple data splits, and out-of-sample validation is generally considered more trustworthy because in-sample errors are prone to overfitting.Source: §6.1 · pp. 12–13

6.2 Optimization Methods

  • The section summarizes automated HPO methods used in forecasting pipelines and introduces approaches for optimizing hyperparameter configurations.The methods optimize configurations using validation data and performance metrics.Source: §6.2 · p. 13, Table 6 · p. 13

6.2.1 Grid Search and Random Search

  • Grid search evaluates every combination in a finite hyperparameter grid, so computation grows exponentially with dimensionality; random search can perform better when only a few parameters matter.Some statistical forecasting procedures combine grid search with preprocessing; Hyndman and Khandakar use a two-stage sARIMA search to reduce evaluations.Source: §6.2.1 · p. 14, Figure 3 · p. 14

6.2.2 Bayesian Optimization

  • Bayesian optimization uses a probabilistic surrogate and an acquisition function to balance exploration and exploitation when selecting the next configuration.Gaussian-process surrogates are recommended for real-valued spaces with expensive training, while random-forest or TPE surrogates suit categorical spaces and conditional choices.Source: §6.2.2 · p. 15, Figure 4 · p. 15

6.2.3 Non-linear Programming

  • Non-linear programming can optimize forecasting hyperparameters when the performance metric is calculable in closed form.Examples include jointly minimizing in-sample RMSE, MAPE, and MAD to select the smoothing parameters and seasonal-cycle length of TES.Source: §6.2.3 · p. 16

6.2.4 Heuristics

  • Heuristics use informed search rules, including ACF- and PACF-based choices of AR and MA lag orders and root finding for the Theta transformation parameter.For the Theta method, Spiliotis et al. compare eight trend and season configurations and select the one with the lowest in-sample MAE.Source: §6.2.4 · p. 16

6.2.5 Metaheuristics

  • Metaheuristics guide searches using feedback from objective values and prior decisions, without requiring assumptions about the objective function.Evolutionary algorithms evaluate populations of candidate configurations and evolve better-performing solutions; genetic algorithms, differential evolution, and estimation-of-distribution algorithms use different update mechanisms.Source: §6.2.5 · p. 16
  • In one MLP comparison, the estimation-of-distribution algorithm had the lowest forecast error after 200 generations, followed by differential evolution and the genetic algorithm [55].Particle swarm optimization has also been used to tune multiple forecasting methods and then combine their best configurations into an ensemble.Source: §6.2.5 · pp. 16–17

6.3 Diagnostic Checking

Diagnostic checking tests whether fitted forecasts and statistical-model assumptions are adequate. The discussion distinguishes HPO strategies by model complexity and recommends integrating residual analysis into pipelines using machine-learning forecasting methods.

6.3 Diagnostic Checking

  • Diagnostic checking assesses residual whiteness, statistically significant relationships, and whether assumed time-series characteristics such as stationarity hold.Adequate residuals have zero mean, finite variance, and no autocorrelation.Source: §6.3 · p. 17
  • Amin, Hwang, and Sekma assess residuals using Box-Pierce or Ljung-Box tests; Amin and Sekma also test parameter significance, while Amin and Hwang check stationarity and invertibility.Amin uses a t-test; its stationarity and invertibility check requires both sums of AR and MA parameters to be smaller than one.Source: §6.3 · p. 17

6.4 Discussion

  • After HPO, diagnostic checking is applied sporadically and only to statistical forecasting methods; the authors recommend integrating automated residual analysis for machine-learning methods.The review notes that residual analysis is not routinely part of machine-learning pipelines.Source: §6.4 · p. 17
  • Differencing orders d and D should be set during pre-processing because differencing makes information-criterion values incomparable across orders; grid search suits low-cost, small ARIMA spaces, while the authors recommend TPE-based Bayesian optimization for complex, large conditional spaces.The latter recommendation concerns methods such as ANNs.Source: §6.4 · p. 17

7 Forecasting Method Selection and Ensembling

The review describes automated forecasting-method selection and forecast ensembling, then discusses how they interact in pipelines. It identifies evidence gaps in selection comparisons and recommends combining selection with ensembling.

7.1 Forecasting Method Selection

  • Fildes and Petropoulos found minimal 1-18-points-ahead out-of-sample MAPE better than 1-point-ahead validation and validation matched to forecast horizon H.Their comparison also considered in-sample MSE and other out-of-sample MAPE horizons.Source: §7.1 · p. 19
  • Selection approaches are heuristic rules, empirical performance comparisons, or decision models relating performance information or meta-features to the selected method.Heuristics use rules based on experience and statistical tests; empirical methods choose by in-sample or validation performance.Source: §7.1 · pp. 18–19
  • Empirical selection requires fitting candidates to target data, whereas selection based only on meta-features avoids that training cost; the latter relies on a trained decision model.Reviewed approaches use meta-feature selection or aggregation and classification methods.Source: §7.1 · p. 20

7.2 Forecast Ensembling

  • Forecast averaging aims to reduce the impact of occasional poor forecasts; averaging all M3 competition submissions outperformed each individual method, and dynamic selection outperformed mean and median ensembling in one experiment.The dynamic approach selects a forecasting method for each forecast origin.Source: §7.2 · pp. 20–21

7.3 Discussion

  • About half of the reviewed work that evaluates multiple methods also ensembles them; the authors recommend combining selection and ensembling, but note that a benchmark comparing selection methods’ compute effort and forecast accuracy is missing.They recommend using selection to determine candidate methods and ensemble weights.Source: §7.3 · p. 21
  • Future work could tailor decision models to specific domains using HPO and test whether pre-trained forecasting methods help on new series with or without retraining.Source: §7.3 · p. 21

8 Automated Forecasting Pipeline

The review maps which forecasting-pipeline sections are covered in the literature and proposes combinations intended to automate the full design process. Most reviewed papers cover only two or three sections, and none cover all five.

8.1 Status Quo

  • Most of the 63 reviewed papers cover only two or three of the five pipeline sections; two cover four, and none cover all five.The review identifies 17 clusters of papers covering the same sections.Source: §8.1 · p. 22

8.2 Discussion

  • The authors recommend combining HPO with feature selection and method selection, and pairing forecast ensembling with method selection; both statistical and machine-learning methods should be considered.The recommendations link feature candidates to their own HPO configurations and use ensembles to eliminate poor candidates.Source: §8.2 · p. 22
  • The review relates hybrid modeling and end-to-end learning to the challenge of addressing all five design sections for high-performing, robust forecasts.It describes end-to-end learning as using one deep neural network to address several sections with appropriate layers.Source: §8.2 · p. 23
  • Manual tailoring remained common in forecasting competitions; the authors cite evidence that an AutoML pipeline achieved competitive performance with moderate computing effort in the M5 competition.Source: §8.2 · p. 23

9 Conclusions

The review finds that most of 63 papers cover only two or three of the five forecasting-pipeline sections, leaving a gap in holistic automation. It examines automation across all five sections and identifies directions for future work.

9 Conclusions

  • The review considers automation in all five pipeline sections and analyzes how AutoML and automated statistical methods interact and combine.Source: §9 · p. 23
  • Most of the 63 reviewed papers cover only two or three of the five forecasting-pipeline sections, leaving a research gap in holistic automation.The authors connect holistic automation with large-scale forecasting without manual, time-consuming tailoring.Source: §9 · p. 23
  • Future work should adapt automation methods for probabilistic forecasts and validate and tailor pipelines to specific use cases before generalizing them.Source: §9 · p. 23
  • The authors also call for performance evaluation, open-source implementations and datasets, and automation of model operation, including monitoring and adaptation.They note performance evaluation could occur in forecasting competitions.Source: §9 · p. 23

Supporting Information

The supporting information gives the search-term expression and reports the staged searches used to identify literature.

Supporting Information

Acronyms

The acronym list expands abbreviations used in the paper, including forecasting and machine-learning terms.

Acronyms

  • AutoML means Automated Machine Learning, while HPO means HyperParameter Optimization.Source: Acronyms · pp. 24–25
  • ARIMA expands to AutoRegressive Integrated Moving Average, and ETS to Error Trend Seasonality.Source: Acronyms · p. 24
  • MAE means Mean Average Error, and RMSE means Root Mean Squared Error.Source: Acronyms · p. 25

Implementations of Forecasting Methods

The appendix lists forecasting methods alongside reported R and Python implementations.

Implementations of Forecasting Methods

Loading 2202.01712v1…