PaperlayerView exact revision on arXiv

Source-linked AI summary

GIFT-Eval: A Benchmark For General Time Series Forecasting Model Evaluation

Taha Aksu, Gerald Woo, Juncheng Liu, Xu Liu, Chenghao Liu, Silvio Savarese, Caiming Xiong, Doyen Sahoo

arXiv:2410.10393v2cs.LGstat.ML

TL;DR

Time series foundation models need a unified, diverse benchmark and non-leaking data for fair zero-shot evaluation. GIFT-Eval addresses this gap with broad evaluation and pretraining components, comprehensive baseline comparisons, and analyses across benchmark characteristics and model behaviors.

  • Problem

    Time series foundation models lack a unified, diverse benchmark and consistent non-leaking data splits for fair comparison.

  • Method

    GIFT-Eval provides separate pretraining and train/test components spanning diverse domains, frequencies, variate counts, prediction lengths, and statistical features.

  • Results

    GIFT-Eval evaluates 17 statistical, deep learning, and foundation-model baselines across benchmark characteristics and qualitative forecasting behaviors.

  • Takeaways & Limitations

    The benchmark supports fair comparison of model families across diverse forecasting settings and exposes their strengths, limitations, and failure cases.

Abstract

from arXiv · show

Time series foundation models excel in zero-shot forecasting, handling diverse tasks without explicit training. However, the advancement of these models has been hindered by the lack of comprehensive benchmarks. To address this gap, we introduce the General Time Series Forecasting Model Evaluation, GIFT-Eval, a pioneering benchmark aimed at promoting evaluation across diverse datasets. GIFT-Eval encompasses 23 datasets over 144,000 time series and 177 million data points, spanning seven domains, 10 frequencies, multivariate inputs, and prediction lengths ranging from short to long-term forecasts. To facilitate the effective pretraining and evaluation of foundation models, we also provide a non-leaking pretraining dataset containing approximately 230 billion data points. Additionally, we provide a comprehensive analysis of 17 baselines, which includes statistical models, deep learning models, and foundation models. We discuss each model in the context of various benchmark characteristics and offer a qualitative analysis that spans both deep learning and foundation models. We believe the insights from this analysis, along with access to this new standard zero-shot time series forecasting benchmark, will guide future developments in time series foundation models. Code, data, and the leaderboard can be found at https://github.com/SalesforceAIResearch/gift-eval .

1 INTRODUCTION

GIFT-Eval addresses the lack of a unified, diverse benchmark for fairly evaluating time series foundation models by combining broad evaluation data with non-leaking pretraining data. It also benchmarks and analyzes statistical, deep learning, and foundation models across varied forecasting characteristics.

  • Motivation: Time series foundation models lack a unified benchmark spanning diverse domains, frequencies, prediction lengths, and variate counts for zero-shot evaluation.Existing evaluations use task-specific datasets with limited diversity, while inconsistent data splits complicate comparisons and risk leakage.
  • GIFT-Eval: GIFT-Eval combines distinct pretraining and train/test components to support fair evaluation of general forecasting capabilities.Its pretraining data are separated from the evaluation data to avoid leakage.
  • GIFT-Eval: 23 datasets contain 144,000 time series and 177 million data points across seven domains, 10 frequencies, short-to-long prediction lengths, and univariate and multivariate settings.The benchmark broadens coverage across forecasting tasks and time series characteristics.
  • Benchmarking and analysis: The benchmark evaluates 17 statistical, deep learning, and foundation-model baselines on diverse forecasting tasks.This comparison is paired with analysis across domains, frequencies, prediction lengths, and numbers of variates.
  • Benchmarking and analysis: The analysis includes qualitative forecasting samples to examine model strengths, forecasting behaviors, and failure cases.The qualitative comparison covers representative transformer-based, traditional deep learning, encoder-decoder, decoder-only, and image-based models.

2 RELATED WORK

Related work spans statistical, deep learning, and foundation-model forecasters, alongside efforts to build broader time series benchmarks. Existing benchmarks provide substantial data or task diversity but remain limited in evaluation variety for foundation models.

  • Forecasting Methods: Time series forecasters include statistical methods, deep learning methods, and newer foundation models.Examples include ARIMA, ETS, Theta, VAR, DeepAR, N-BEATS, DLinear, transformer-based models, Moirai, Chronos, and TimesFM.
  • Forecasting Benchmarks: LOTSA contains 231 billion data points across nine domains but reuses existing evaluation benchmarks with limited variety in data characteristics and forecasting tasks.GIFT-Eval is presented as augmenting this evaluation coverage.

3 GIFT-EVAL

GIFT-Eval defines forecasting tasks and benchmark characteristics, then builds a diverse evaluation and pretraining resource whose distributions are analyzed across domains, frequencies, horizons, variate counts, and statistical features.

  • 3.1.1 TIME SERIES FORECASTING: Time series forecasting predicts future values for one or more variates from regularly sampled historical data, optionally using covariates.The task is represented through a predictive distribution conditioned on a context window and forecast horizon.
  • 3.1.2 TIME SERIES CHARACTERISTICS AND FEATURES: GIFT-Eval targets diversity across domain, observation frequency, prediction length, and number of variates to represent varied forecasting scenarios.Prediction length denotes the number of future steps, while variate count captures dimensionality and cross-series dependencies.
  • 3.1.2 TIME SERIES CHARACTERISTICS AND FEATURES: Six statistical features measure temporal attributes, forecastability, and regularity, including trend, seasonal strength, entropy, Hurst exponent, stability, and lumpiness.Trend and seasonal strength describe temporal patterns; entropy and Hurst exponent assess forecastability-related properties, while stability and lumpiness characterize variation and regularity.
  • 3.2 DATASETS: The benchmark’s test data are organized quantitatively by prediction length, domain, frequency, and variate count, with additional dataset-level statistics available.These breakdowns provide a quantitative overview of the test benchmark’s composition.
  • 3.2 DATASETS: Figure 1’s heatmaps compare mean feature values across dataset characteristics, revealing distinct patterns by variate count, horizon, domain, and frequency.Multivariate series show higher stability and lumpiness, longer horizons show lower trend and Hurst values, and yearly or quarterly data show stronger persistence-related patterns.
  • 3.2 DATASETS: GIFT-Eval includes a non-leaking pretraining resource spanning 71 univariate and 17 multivariate datasets, seven domains, 13 frequencies, 4.5 million series, and 230 billion data points.The aligned pretraining collection is intended for models evaluated on the benchmark without overlap between pretraining and evaluation data.

4 EXPERIMENTS

GIFT-Eval evaluates forecasting models across domains, horizons, frequencies, and variate settings using normalized MAPE, CRPS, and rank metrics. Results show that model strengths depend on benchmark characteristics, while qualitative examples expose forecasting failures.

  • Evaluation setting: Performance is aggregated by domain, prediction length, frequency, number of variates, and all benchmark configurations.The evaluation uses MAPE for point forecasts, CRPS for probabilistic forecasts, and rank-based aggregation, with results organized across these benchmark dimensions.
  • Domain: Foundation models generally lead across domains, but PatchTST and TFT perform best in Web/CloudOps and Transport, where series are less predictable or more lumpy.The authors associate inconsistent foundation-model performance across domains with differences in pretraining datasets and their contribution ratios.
  • Prediction length: Moirai variants lead short-term forecasts, whereas PatchTST performs better at medium and long horizons as TimesFM and Chronos decline.The paper attributes the latter decline to recursive multi-step forecasting and accumulated error, while noting a remaining gap between deep learning and foundation models at longer horizons.
  • Frequency: Auto_Theta leads at the highest frequency, deep learning models dominate many minutely and hourly settings, and foundation models lead from daily through yearly frequencies.Higher-frequency data is described as noisier and less pattern-rich, whereas lower-frequency data contains broader patterns and slower dynamics.
  • Number of variates: PatchTST and iTransformer are strongest in multivariate settings, while large Moirai performs best among foundation models and leads in univariate scenarios.Moirai is identified as the only foundation model in this comparison that supports multivariate forecasting.
  • General and qualitative results: PatchTST has the best average scores across the full benchmark, with MoiraiLarge second, but GIFT-Eval does not consistently confirm that larger foundation models perform better.The reported scaling pattern holds only in the energy domain or for univariate forecasting when judged with ranking metrics; qualitative plots also show model-specific failures.

5 CONCLUSION

GIFT-Eval is a comprehensive, diverse benchmark for fairly evaluating zero-shot time series forecasting models and accelerating foundation-model development.

  • GIFT-Eval evaluates forecasting models across diverse domains, frequencies, variate counts, prediction lengths, and statistical characteristics.It includes non-leaking pretraining data, 17 baselines, model-strength analysis, and qualitative failure-case analysis.

A EXPERIMENTAL SETUP DETAILS

The experimental setup evaluates statistical, deep learning, and foundation models with specified implementations, resources, and time-series feature calculations.

  • Statistical models: Five statistical baselines use statsforecast on a 96-core CPU server with one day per dataset and Seasonal Naive fallback after timeout.
  • Deep learning models: Deep learning baselines use GluonTS models or custom wrappers, with probabilistic heads where feasible and hyperparameter searches across runs.
  • Foundation models: Foundation models are evaluated zero-shot on the benchmark test split using their publicly available implementations.Moirai uses frequency-to-patch-size mapping; TimesFM and VisionTS follow their stated evaluation settings.
  • Additional parameters and computational resources: All experiments use eight NVIDIA A100 GPUs, while trained models use original losses and optimizers, batch size 128, 100 batches per epoch, and 50 epochs.
  • Time series features: The benchmark analyzes six time-series properties, computes features using recent histories of up to 500 points, and excludes NaN values during aggregation.Prediction lengths remain faithful to each dataset and are not clipped.
  • Time series features: STL decomposes each series into trend, seasonal components, and remainder; trend and seasonal strengths quantify explained variability and are clipped to [0,1].
  • Time series features: Lower spectral entropy indicates higher signal-to-noise ratio and more predictable patterns, whereas higher entropy indicates greater complexity and forecasting difficulty.
  • Time series features: Higher stability indicates greater differences in tiled means, while higher lumpiness indicates more variation in tiled variances and volatility.

C EVALUATION METRICS

GIFT-Eval measures point and probabilistic forecasting performance using MAPE and CRPS, with CRPS approximated through weighted quantile losses.

  • MAPE: MAPE evaluates point forecasting as the mean absolute percentage difference between actual and forecasted values.It is scale-independent and easy to interpret but is sensitive to zero or near-zero actual values.
  • CRPS: CRPS evaluates probabilistic forecasts by comparing predicted cumulative distribution functions with observed values.
  • CRPS: CRPS is approximated by a discrete sum of weighted quantile losses over selected quantile levels.K denotes the number of quantile levels, and α1 through αK denote the selected levels.
  • CRPS: The weighted quantile loss uses predicted α-quantiles, observed values, and quantile loss at each time step.

D GIFT-EVAL TEST DATASETS

GIFT-Eval’s test portion combines datasets from 10 open-domain sources covering weather, business and IT observability, restaurants, and electricity forecasting.

  • Dataset sources and properties: The test benchmark includes Jena Weather, BizITObs, Bitbrains, Restaurant, ETT1, and ETT2 datasets from multiple application domains.BizITObs fuses business KPIs and IT event channels into multivariate series, while Restaurant forecasting predicts future visitor totals.

E GIFT-EVAL PRE-TRAINING DATASETS

The pre-training split is built from LOTSA-derived datasets, with selected exclusions creating a more diverse and balanced evaluation arrangement. Its sources span energy, climate, cloud operations, load forecasting, and broad multivariate time-series collections.

  • GIFT-Eval’s pre-training split is constructed from LOTSA, with selected datasets excluded to diversify and balance evaluation.
  • The pre-training sources cover building energy, climate variables, CloudOps utilization, load forecasting, and diverse general time-series datasets.
  • The complete pre-training dataset list and key properties are provided in Table 14.

F.1 RESULTS WITH ALL MODELS

The appendix reports results for all evaluated models, including models omitted from the main paper, using aggregated and non-aggregated views across benchmark configurations.

  • Results for all models, including those omitted from the main paper, are reported in aggregated tables.
  • Aggregated results are organized in Tables 15 to 19.
  • Non-aggregated results cover dataset, term-length, and frequency combinations in Tables 20 to 22.

F.2 DATA LEAKAGE EFFECT IN FOUNDATION MODELS

Different foundation models can be affected by partial pre-training-data leakage on GIFT-Eval, so the paper compares leakage-controlled and original Moirai models. Qualitative examples also show that model performance varies across datasets and forecasting settings.

  • Data leakage: Partial data leakage affects evaluation because foundation models use different pre-training datasets.
  • Data leakage: The authors pre-trained new Moirai models on GIFT-Eval pre-training data and compare them with original Moirai models labeled Moirai-Leakage.
  • Data leakage: Leakage from training sets can improve corresponding test-set performance, especially as prediction length increases.
  • Qualitative examples: On Bizitobs_l2c, Chronos degrades over longer horizons, Moirai misses regular peaks and troughs, and VisionTS is closest to the ground truth.
  • Qualitative examples: On the high-frequency Electricity dataset, Chronos is more consistent than Moirai and VisionTS, while Moirai predicts stationary changes misaligned with the ground truth.
Loading 2410.10393v2…