PaperlayerView exact revision on arXiv

Source-linked AI summary

Do We Really Need Deep Learning Models for Time Series Forecasting?

Shereen Elsayed, Daniela Thyssens, Ahmed Rashed, Hadi Samer Jomaa, Lars Schmidt-Thieme

arXiv:2101.02118v2cs.LGstat.ML

TL;DR

The paper examines whether simple GBRT models need deep learning to achieve competitive time-series forecasts. It converts forecasting into window-based regression, engineers GBRT inputs and outputs, and finds that the resulting model competes with and sometimes outperforms state-of-the-art DNN models, although reproducibility requirements exclude some relevant approaches.

  • Problem

    The paper asks whether increasingly complex deep learning forecasting models are necessary when simpler models may be competitive with suitable configuration.

  • Method

    The study compares a feature-engineered, multi-output GBRT using window-based inputs with selected deep learning forecasting models across univariate and multivariate tasks.

  • Results

    The configured GBRT competes with and sometimes outperforms state-of-the-art DNN models in the evaluated forecasting comparisons.

  • Takeaways & Limitations

    Simple machine-learning baselines should be carefully configured rather than dismissed when assessing progress in time-series forecasting.

  • Takeaways & Limitations

    The evaluation excludes some conceptually relevant approaches because publicly available datasets or source code were unavailable.

Abstract

from arXiv · show

Time series forecasting is a crucial task in machine learning, as it has a wide range of applications including but not limited to forecasting electricity consumption, traffic, and air quality. Traditional forecasting models rely on rolling averages, vector auto-regression and auto-regressive integrated moving averages. On the other hand, deep learning and matrix factorization models have been recently proposed to tackle the same problem with more competitive performance. However, one major drawback of such models is that they tend to be overly complex in comparison to traditional techniques. In this paper, we report the results of prominent deep learning models with respect to a well-known machine learning baseline, a Gradient Boosting Regression Tree (GBRT) model. Similar to the deep neural network (DNN) models, we transform the time series forecasting task into a window-based regression problem. Furthermore, we feature-engineered the input and output structure of the GBRT model, such that, for each training window, the target values are concatenated with external features, and then flattened to form one input instance for a multi-output GBRT model. We conducted a comparative study on nine datasets for eight state-of-the-art deep-learning models that were presented at top-level conferences in the last years. The results demonstrate that the window-based input transformation boosts the performance of a simple GBRT model to levels that outperform all state-of-the-art DNN models evaluated in this paper.

1 Introduction

The paper asks whether carefully configured GBRT models can match or exceed deep learning approaches for univariate and multivariate time-series forecasting. It frames input and output feature engineering as central to making a simple model competitive.

  • Carefully configured GBRT can compete with and sometimes outperform state-of-the-art deep neural forecasting models.The model is evaluated on both univariate and multivariate forecasting tasks.
  • The study addresses whether GBRT input and output configuration affects performance in a window-based forecasting framework.The research questions explicitly separate configuration effects from comparison with deep learning frameworks.
  • The comparison includes state-of-the-art deep learning models selected from prominent research conferences and covers univariate and multivariate tasks.The experimental design evaluates GBRT against models featured at top research conferences.
  • The proposed GBRT casts forecasting as window-based regression and feature-engineers inputs and outputs to exploit additional context information.The target configuration is designed to bring GBRT closer to the input handling used by successful forecasting models.
  • The study also compares window-based GBRT with traditionally configured ARIMA and naive GBRT implementations.This comparison is intended to assess the contribution of the window-based input setting.

2 Research Design

The research design systematically selects reproducible forecasting baselines and evaluates them with GBRT on shared datasets across univariate and multivariate tasks. Some conceptually relevant methods are excluded because their data or implementations cannot be reproduced.

  • 2.1 Collecting Baseline Papers: Baseline papers are filtered from major conferences using topic, data-structure, reproducibility, and computational-feasibility requirements.The selection targets recent time-series forecasting work from 2016 to 2020.
  • 2.1 Collecting Baseline Papers: Several conceptually suitable approaches are excluded because source code or publicly available datasets are missing.The excluded works are listed in Appendix A.
  • 2.2 Evaluation: The evaluation compares configured GBRT and selected deep learning baselines on the same pool of univariate and multivariate datasets.The dataset statistics and experimental specifications are summarized in Table 1.
  • 2.2 Evaluation: Some Electricity and Traffic datasets are sub-sampled, with additional head-to-head comparisons used to validate findings under affected settings.Certain baselines are re-evaluated and re-tuned under the changed specifications.
  • 2.2 Evaluation: The study uses datasets including Traffic, Electricity, PeMSD7(M), Exchange Rate, Solar Energy, Beijing PM2.5, and Urban Air Quality.Table 1 records dataset and experimental setup information.

3 Problem Formulation

The paper formulates time-series forecasting as supervised learning over sequence-valued predictors and targets. It distinguishes univariate forecasting without covariates from multivariate forecasting with auxiliary information.

  • Time-series forecasting is formulated as learning a function from sequence-valued predictors to target sequences while minimizing expected loss.The formulation specifies predictor and target spaces, a data sample, and a loss function.
  • The predictors consist of sequences of vector pairs, while the target is represented as a target sequence.This formulation supports the forecasting cases studied in the paper.
  • Univariate forecasting uses one target channel and no additional covariates, so predictors contain only target-channel vectors.In the notation, this corresponds to L = 1 and M = 0.
  • Multivariate forecasting is treated as a separate special case involving target channels and auxiliary information.The supplied formulation introduces this case after defining the univariate setting.

4 Feature-engineered window-based GBRT

The paper reformulates time-series forecasting as window-based regression for GBRT, flattening windowed inputs and using a multi-output configuration to predict future horizons.

  • 4 Feature-engineered window-based GBRT: Window-based GBRT reshapes time-series windows into flattened vectors before training on multiple window instances.The transformation maps windowed inputs with target channels and covariates into GBRT-suitable one-dimensional vectors.
  • 4 Feature-engineered window-based GBRT: The model appends the final time-point covariates and passes the resulting vectors to a multi-output GBRT for future-horizon prediction.The horizon contains h future time steps, with a separate regressor introduced for each prediction step through a multi-output wrapper.
  • 4 Feature-engineered window-based GBRT: The configuration emphasizes window-based input handling because independently forecasted horizon targets cannot exploit relationships among future outputs.The multi-output wrapper transforms the problem into several single-target regressions, while the window formulation supplies the central modeling adaptation.

5 Experiments and Results

The experiments compare reconfigured GBRT with prominent deep-learning and matrix-factorization forecasters across univariate and multivariate settings, alongside naive GBRT and ARIMA baselines.

  • 5 Experiments and Results: The study evaluates acclaimed DNN approaches against a reconfigured GBRT baseline across separate univariate and multivariate forecasting experiments.The documented results use final runs that include validation data as training data.
  • 5 Experiments and Results: The evaluated approaches include TRMF, LSTNet, DARNN, DeepAR, DeepState, and DAQFF, representing matrix-factorization, recurrent, attention-based, and convolutional architectures.The supplied model descriptions identify distinct mechanisms such as global structure modeling, convolutional and recurrent layers, attention, and probabilistic state-space modeling.
  • 5 Experiments and Results: Naive GBRT and ARIMA are included to test whether casting forecasting as rolling window-based regression improves over conventional input handling.The naive baselines fit the complete consecutive training portion before evaluating the remaining test portion.
  • 5 Experiments and Results: Univariate forecasting predicts one future target from that target’s history, while multiple independent single-target series are not treated as multivariate forecasting.The study also allows simple time covariates to be constructed for the forecasting task.

Results for Uni-variate Time Series

Across univariate datasets, window-based GBRT is highly competitive and generally surpasses naively configured baselines, although traffic forecasting remains an exception and some metrics favor other models.

  • Results for Uni-variate Time Series: Window-based GBRT shows strong overall competitiveness on univariate datasets, while ARIMA and naive GBRT are outperformed by far.The reported gains in the no-covariate setting are attributed exclusively to the rolling forecasting formulation.
  • Results for Uni-variate Time Series: For electricity forecasting, window-based GBRT achieves the best RMSE across all models, while TRMF alone surpasses it on WAPE and MAE.This comparison is reported for the univariate evaluation without covariates.
  • Results for Uni-variate Time Series: For exchange-rate prediction, window-based GBRT outperforms the comparably strong LSTNet and TRMF results.LSTNet is reported with a forecasting window of w = 24 in this comparison.
  • Results for Uni-variate Time Series: Traffic results are mixed: DARNN and LSTNet lead on hourly Traffic, whereas reconfigured GBRT outperforms STGCN on PeMSD7.The cited PeMSD7 comparison reports STGCN’s RMSE as 6.77 before stating that the GBRT baseline outperformed it.
  • Results for Uni-variate Time Series: Additional evaluations with time covariates reinforce that windowing inputs and adding simple covariates generalize convincingly across univariate datasets.The authors describe these experiments as further affirmation of the broader finding across Tables 2 and 3.

Complementary Comparison against LSTNet

On the Solar-Energy and Exchange-Rate comparisons, window-based GBRT remained competitive with LSTNet, while broader probabilistic and transformer comparisons showed a qualified advantage over several deep models.

  • LSTNet comparison: Window-based GBRT consistently outperformed LSTNet in the complementary Solar-Energy comparison.The comparison used time covariates, a forecasting window of h = 24, RSE, and Corr.
  • Probabilistic and transformer models: In the broader univariate evaluation, TFT outperformed GBRT, whereas DeepAR and DeepState were outperformed on the evaluated datasets.The results therefore distinguish TFT from the probabilistic models in this comparison.
  • Feature effects: Timestamp-derived covariates substantially elevated GBRT performance, motivating extension to more expressive dataset features.This finding connects the complementary comparisons to the subsequent multivariate experiments.

5.3 Multivariate Datasets

The multivariate setting uses one target series conditioned on externally supplied features, with experiments comparing GBRT against models designed for such forecasting tasks.

  • Multivariate setting: Multivariate forecasting here predicts one target variable while conditioning on multiple externally provided features.The paper distinguishes this setting from forecasting multiple time-series instances.
  • Feature design: The study uses expressive external features rather than only simplistic predictors extracted from timestamps.The multivariate experiments evaluate two task-specific forecasting comparisons: DARNN and DAQFF against GBRT.
  • Experimental tasks: The DARNN comparison predicts room temperature and stock price one step ahead using a lookup window of 10 data points.These tasks use various predictive features and follow the lookup-window setting identified as best for DARNN.

Comparison against DARNN with Covariates

Across multivariate comparisons, well-configured GBRT outperformed DARNN and DAQFF-related alternatives, including a task-specific air-quality DNN in the reported experiments.

  • DARNN comparison: Simple, well-configured GBRT outperformed the attention-based DARNN framework in the multivariate comparison.The result supports the preceding univariate finding that careful GBRT configuration can challenge specialized DNN baselines.
  • Evaluation implications: The comparisons show that baseline configuration and baseline selection both matter when evaluating machine-learning forecasting models.The authors specifically caution against relying only on ARIMA as the non-DNN comparator.
  • DAQFF comparison: For air-quality forecasting, the feature-engineered GBRT outperformed DAQFF, while even traditional GBRT performed better than DAQFF in the reported experiment.DAQFF was specifically designed for air-quality prediction, but its original results were not reproducible because source code was unavailable.

5.4 Ablation Study

The ablation study finds that retaining only the latest windowed instance’s auxiliary features preserves competitive performance while reducing computational demands.

  • Feature inclusion ablation: Using only the last instance’s auxiliary features caused barely any information loss compared with using covariates from all windowed instances.The result supports a reduced feature-inclusion scheme for the window-based GBRT.

6 Conclusion

The study finds that carefully feature-engineered GBRT can match or outperform state-of-the-art deep-learning forecasting models, supporting simpler baselines in comparative evaluation.

  • GBRT can compete with and sometimes outperform state-of-the-art DNN models when its input and output structures are efficiently feature-engineered.The comparison used reproduced deep-learning frameworks and rolling-forecast GBRT across various datasets.
  • These findings suggest that simpler machine-learning baselines should be configured carefully rather than dismissed when assessing progress in time-series forecasting.
  • The window-based input setting could be extended to other simpler models, including multilayer perceptrons and support vector machines.
Loading 2101.02118v2…