PaperlayerView exact revision on arXiv

Source-linked AI summary

TFB: Towards Comprehensive and Fair Benchmarking of Time Series Forecasting Methods

Xiangfei Qiu, Jilin Hu, Lekui Zhou, Xingjian Wu, Junyang Du, Buang Zhang, Chenjuan Guo, Aoying Zhou, Christian S. Jensen, Zhenli Sheng, Bin Yang

arXiv:2403.20150v4cs.LGcs.AIcs.CY

TL;DR

TSF research needs comprehensive and reliable empirical comparisons across diverse domains and methods. TFB addresses this gap with broad datasets, method coverage, and a flexible evaluation pipeline, finding that method performance varies with dataset characteristics and that traditional methods can outperform recent SOTA methods on some datasets.

  • Problem

    Existing TSF evaluations provide insufficient domain coverage, can bias comparisons against traditional methods, and use inconsistent or inflexible pipelines.

  • Method

    TFB is an automated benchmark combining diverse datasets, statistical-learning, machine-learning, and deep-learning methods, multiple evaluation strategies and metrics, and a flexible scalable pipeline.

  • Results

    TFB evaluations show that method performance depends on dataset characteristics, with VAR and LinearRegression outperforming recent SOTA methods on some datasets.

  • Takeaways & Limitations

    Comprehensive, standardized benchmarking supports fairer and more meaningful comparisons of TSF methods across domains and settings.

  • Takeaways & Limitations

    Rolling forecasting uses retraining for statistical methods but reinferring for machine-learning and deep-learning methods to balance runtime and prediction accuracy.

Abstract

from arXiv · show

Time series are generated in diverse domains such as economic, traffic, health, and energy, where forecasting of future values has numerous important applications. Not surprisingly, many forecasting methods are being proposed. To ensure progress, it is essential to be able to study and compare such methods empirically in a comprehensive and reliable manner. To achieve this, we propose TFB, an automated benchmark for Time Series Forecasting (TSF) methods. TFB advances the state-of-the-art by addressing shortcomings related to datasets, comparison methods, and evaluation pipelines: 1) insufficient coverage of data domains, 2) stereotype bias against traditional methods, and 3) inconsistent and inflexible pipelines. To achieve better domain coverage, we include datasets from 10 different domains: traffic, electricity, energy, the environment, nature, economic, stock markets, banking, health, and the web. We also provide a time series characterization to ensure that the selected datasets are comprehensive. To remove biases against some methods, we include a diverse range of methods, including statistical learning, machine learning, and deep learning methods, and we also support a variety of evaluation strategies and metrics to ensure a more comprehensive evaluations of different methods. To support the integration of different methods into the benchmark and enable fair comparisons, TFB features a flexible and scalable pipeline that eliminates biases. Next, we employ TFB to perform a thorough evaluation of 21 Univariate Time Series Forecasting (UTSF) methods on 8,068 univariate time series and 14 Multivariate Time Series Forecasting (MTSF) methods on 25 datasets. The benchmark code and data are available at https://github.com/decisionintelligence/TFB. We have also launched an online time series leaderboard: https://decisionintelligence.github.io/OpenTS/OpenTS-Bench/.

1 INTRODUCTION

TFB addresses the need for fairer and more comprehensive empirical comparisons of TSF methods by expanding dataset coverage, method coverage, and evaluation support. Its experiments examine how forecasting performance varies across dataset characteristics and methods.

  • Motivation: TFB addresses three evaluation issues: limited domain coverage, bias against traditional methods, and inflexible comparison pipelines.Existing benchmarks do not jointly provide comprehensive coverage across these properties.
  • Motivation: Time series from different domains exhibit distinct and complex characteristics, including seasonality, trends, abrupt changes, and combinations of patterns.Examples include environmental temperature seasonality, economic growth trends, and electricity-series shifts.
  • Motivation: Existing multivariate benchmarks cover at most 5 domains and concentrate mainly on traffic and electricity datasets.TSlib, LTSF-Linear, BasicTS, and BasicTS+ each include around 10 datasets or fewer.
  • TFB: TFB provides taxonomically organized datasets, diverse statistical, machine-learning, and deep-learning methods, and multiple evaluation strategies and metrics.Its unified pipeline standardizes datasets and evaluation procedures to support fairer comparisons.
  • Findings: VAR and LinearRegression outperform recently proposed state-of-the-art methods on some datasets, while method performance depends on dataset characteristics.Linear methods perform well with increasing trends or shifts, whereas transformer methods perform better with seasonality and nonlinear patterns.
  • TFB: TFB evaluates more than 20 UTSF methods on 8,068 univariate series and 14 MTSF methods on 25 multivariate datasets.The benchmark also provides an automated end-to-end workflow and an online leaderboard.

2 RELATED WORK

TSF methods span statistical learning, machine learning, and deep learning, while existing benchmarks often cover only one forecasting setting or method paradigm. TFB addresses these gaps with broader dataset characterization and a scalable pipeline intended to support fairer comparisons.

  • TSF methods: TSF methods comprise statistical learning, machine learning, and deep learning approaches.Classical methods include ARIMA, ETS, Theta, VAR, and Kalman Filter; later work includes tree-based and neural methods.
  • Benchmark coverage: Most earlier benchmarks target either univariate or multivariate forecasting, while only TSlib considers both.
  • Benchmark coverage: Benchmark method diversity is limited in opposite directions: early studies often omit deep learning, whereas recent benchmarks frequently focus solely on deep learning.
  • Benchmark pipelines: Existing benchmark pipelines are either absent or specialized for statistical or deep learning methods, limiting integration and scalability.M3 and M4 provide no pipeline; Monash and Libra target statistical methods, while LTSF-Linear and TSlib target deep learning methods.
  • TFB: TFB aims to provide broader method coverage and a more scalable implementation for reliable, thorough, and user-friendly evaluation.

3 PRELIMINARIES

The preliminaries define forecasting and characterize time series using trend, seasonality, stationarity, shifting, transition, and correlation. These characteristics support fine-grained dataset analysis and more selective matching of forecasting methods to scenarios.

  • Characteristics: The benchmark characterizes datasets through trend, seasonality, stationarity, shifting, transition, and correlation.Trend and seasonality capture temporal patterns; stationarity concerns distributional properties, while shifting, transition, and correlation describe additional dataset behavior.
  • Definitions: A time series is a chronological sequence of N-dimensional time points, classified as univariate when N=1 and multivariate when N>1.
  • Definitions: Time series forecasting predicts the next F future points from H historical points, with F denoting the forecasting horizon.
  • Characteristics: Shifting measures changes in a time series probability distribution over time, with values approaching 1 indicating more severe shifting.
  • Characteristics: Transition measures the trace of the covariance of a transition matrix over a 3-letter alphabet and captures regular, identifiable fixed features.
  • Use of characterization: The defined characteristics reveal differences among real-world series and support more selective selection of forecasting methods for specific scenarios.

4 TFB: BENCHMARK DETAILS

TFB combines consistently formatted datasets with characterization-based coverage analysis to support broad empirical evaluation of forecasting methods. Its collection spans 25 multivariate and 8,068 univariate datasets, with diversity assessed across domains and time-series characteristics.

  • 4 TFB: BENCHMARK DETAILS: The benchmark design covers dataset collection, supported methods, evaluation strategies and metrics, and the complete benchmarking pipeline.These components are described in Sections 4.1 through 4.4.
  • 4 TFB: BENCHMARK DETAILS: TFB organizes 25 multivariate and 8,068 univariate datasets in a consistently formatted collection spanning diverse domains and characteristics.The collection is intended to address varied formats, documentation, and dataset-collection effort.
  • 4 TFB: BENCHMARK DETAILS: The univariate datasets are curated from 16 open-source datasets and analyzed with Principal Feature Analysis to preserve original time-series values.PFA is used as a variation of PCA for representing the complexity of real-world series.
  • 4 TFB: BENCHMARK DETAILS: TFB characterizes univariate series using trend, seasonality, stationarity, shifting, and transition, then reduces these five indicators to two dimensions with PCA for visualization.The resulting distributions are shown as hexbin plots for the eight most widely distributed univariate datasets.
  • 4 TFB: BENCHMARK DETAILS: TFB and M4 cover the most characteristic-distribution cells, while TFB covers a wider range of domains than M4 with about 8% as many experiments.M4 contains 100,000 series, compared with TFB’s 8,068.

4.2 Comparison Methods

TFB compares forecasting methods across statistical learning, machine learning, and deep learning categories. Its comparison set includes established statistical and machine-learning baselines alongside deep-learning methods.

  • 4.2 Comparison Methods: TFB includes 22 methods spanning statistical learning, machine learning, and deep learning categories.The listed statistical methods include ARIMA, ETS, Kalman Filter, and VAR; machine-learning methods include XGBModel, LinearRegression, and Random Forest.
  • 4.2 Comparison Methods: The statistical-learning comparison set contains ARIMA, ETS, Kalman Filter, and VAR.
  • 4.2 Comparison Methods: The machine-learning comparison set contains XGBModel, LinearRegression, and Random Forest, alongside the benchmark’s deep-learning methods.

4.3 Evaluation Settings

TFB evaluates forecasting accuracy with fixed and rolling strategies, using standardized horizon-based predictions and multiple error metrics. Rolling evaluation averages metric values across forecasting windows and accommodates different retraining practices across method families.

  • 4.3 Evaluation Settings: TFB implements fixed forecasting and rolling forecasting as two distinct evaluation strategies.Fixed forecasting predicts f future points from n−f historical points, while rolling forecasting repeatedly advances across windows.
  • 4.3 Evaluation Settings: In rolling forecasting, each inference iteration uses historical data to predict the designated forecasting horizon, after which error metrics are averaged across iterations.
  • 4.3 Evaluation Settings: Statistical-learning methods commonly retrain on all or part of the historical data for each rolling forecast, whereas machine-learning and deep-learning methods use different retraining practices.The benchmark discusses retraining choices in relation to runtime and prediction timeliness.
  • 4.3 Evaluation Settings: TFB uses eight error metrics: MAE, MAPE, MSE, SMAPE, RMSE, WAPE, MSMAPE, and MASE.The metrics are defined in Equations 7–14; the parameter ε in Equation 13 is set to 0.1.
  • 4.3 Evaluation Settings: For rolling forecasting, TFB averages error metrics across all samples or windows on each time series to assess method performance.

4.4 Unified Pipeline

TFB uses a unified pipeline with data, method, evaluation, and reporting layers to standardize benchmarking. The pipeline supports diverse methods, evaluation configurations, integrations, and computing environments.

  • 4.4 Unified Pipeline: TFB’s unified pipeline consists of data, method, evaluation, and reporting layers.The structure is designed to reduce the effect of implementation differences on evaluation results.
  • 4.4 Unified Pipeline: The data layer stores univariate and multivariate series from diverse domains using standardized formats and characteristic-based organization.It can assess whether a new dataset expands the distribution of existing datasets across six features.
  • 4.4 Unified Pipeline: The method layer supports statistical-learning, machine-learning, and deep-learning methods, third-party TSF libraries, and both DMS and IMS forecasting.Third-party methods can be integrated through a Universal Interface.
  • 4.4 Unified Pipeline: The evaluation layer supports customized metrics, standardized dataset handling, splitting, normalization, and configurable evaluation settings.
  • 4.4 Unified Pipeline: Users deploy a method architecture and select or configure a file, after which TFB automatically runs the benchmark pipeline.The pipeline also supports CPU and GPU hardware plus sequential and parallel execution.
  • 4.4 Unified Pipeline: TFB is presented as a unified, flexible, scalable, and user-friendly tool for understanding, comparing, and selecting TSF methods for application scenarios.

5 EXPERIMENTS

TFB evaluates forecasting methods across the datasets and methods covered in the benchmark, using comprehensive hyperparameter selection to support fair comparison.

  • TFB evaluates 14 multivariate and 22 univariate forecasting methods on all covered datasets.The experiments use the pipeline described in Section 4.4.

5.1 Experimental Setup

The experimental setup includes all TFB datasets and methods, tests multiple forecasting horizons and look-back windows, searches hyperparameters, and reports selected metrics alongside reproducibility resources.

  • The experiments include all datasets and methods covered by TFB.
  • Multivariate forecasting uses rolling forecasts with dataset-dependent horizons and look-back windows.Horizons and windows differ between shorter and longer datasets.
  • Each method uses its original hyperparameters plus searches over up to 8 additional sets, selecting the optimal result.This selection is intended to support comprehensive and unbiased assessment.
  • Only a subset of metrics is reported in the paper, with additional results available in the code repository.
  • Experiments use PyTorch with L2 loss and ADAM on an NVIDIA Tesla-A800 GPU, and datasets and code are provided for reproducibility.Batch size starts at 32 and can be reduced to 8 under out-of-memory conditions.

5.2 Experimental Results

Results show that forecasting performance depends on method class and dataset characteristics: deep learning often performs strongly on average, while traditional and machine-learning methods can lead in particular settings.

  • 5.2.1 Univariate time series forecasting: Deep methods including TimesNet, PatchTST, and N-HiTS achieve substantially better average UTSF MASE and MSMAPE.LR and RF nevertheless outperform competitors by MSMAPE rank.
  • 5.2.1 Univariate time series forecasting: LR performs better on seasonal, trending, and shifting series, whereas RF performs better when those patterns are absent.LR is also more suitable for nonstationary data than RF.
  • 5.2.2 Multivariate time series forecasting: No single multivariate method achieves the best performance on every dataset.Transformer methods generally perform better on weak-trend datasets, while Linear methods tend to perform moderately better on strong-trend datasets.
  • 5.2.2 Multivariate time series forecasting: Recent methods do not consistently outperform earlier methods such as Informer, LR, and VAR.The results support evaluating methods across a broader range of datasets.
  • 5.2.3 Performance on different characteristics: No deep learning method excels across all six dataset characteristics.Crossformer, PatchTST, and NLinear each achieve their strongest results on different characteristic-focused datasets.

5.3 Hints to Method Design

The design analysis links method choice to dataset characteristics, channel dependencies, and computational cost, showing that different architectures have distinct strengths and trade-offs.

  • 5.3.1 Method performance across characteristics: CNN, Linear, and Transformer methods exhibit distinct advantages on datasets with different characteristics.
  • 5.3.1 Method performance across characteristics: Transformer-based, CNN-based, and Linear-based methods should be compared according to the time-series characteristics they address.Figure 9 marks the methods with minimum MAE using red triangles.
  • 5.3.1 Method performance across characteristics: Method architectures can flexibly adapt to complex time-series patterns and intrinsic correlations.
  • 5.3.1 Method performance across characteristics: Selecting a method based on the relevant time-series characteristics can help leverage its strengths.
  • 5.3.2 Channel dependency: As within-dataset correlations increase, Crossformer gradually surpasses PatchTST, indicating a benefit from modeling channel dependencies.The comparison uses MAE at forecasting horizon 96 across ten datasets.
  • 5.3.3 Running time and parameter: Linear-based methods outperform CNN-based and Transformer-based methods when inference time and parameter counts are considered.Inference time generally increases with parameter count; PatchTST is faster than Triformer and Crossformer among Transformer methods.

6 CONCLUSIONS

TFB is designed as a comprehensive and fair benchmark for reliable comparison of time series forecasting methods. It broadens dataset coverage, method diversity, evaluation options, and pipeline flexibility.

  • TFB covers time series datasets from 10 domains to address insufficient data-domain coverage.The domains include traffic, electricity, energy, environment, nature, economics, stock, banking, health, and web.
  • A time series characteristics analysis helps ensure that selected datasets are distributed across different characteristics.
  • TFB includes statistical learning, machine learning, and deep learning methods with varied evaluation strategies and metrics.
  • TFB provides a flexible and scalable pipeline intended to eliminate biases from inconsistent and inflexible evaluation procedures.
  • Together, these design choices aim to support more comprehensive and fair performance comparisons and promote TSF method development.
Loading 2403.20150v4…