PaperlayerView exact revision on arXiv

Source-linked AI summary

Monash Time Series Forecasting Archive

Rakshitha Godahewa, Christoph Bergmeir, Geoffrey I. Webb, Rob J. Hyndman, Pablo Montero-Manso

arXiv:2105.06643v1cs.LGstat.ML

TL;DR

Global forecasting models require benchmarks containing related time series, but comprehensive archives for this purpose were unavailable. The paper presents an archive with dataset formats and feature analysis, evaluates baseline models across datasets, and identifies dataset differences while documenting computational and metric-scope limitations.

  • Problem

    Comprehensive forecasting archives for evaluating global models on related time series were unavailable, despite the importance of forecasting for business and industry decisions.

  • Method

    The paper constructs a 20-dataset archive, introduces a flexible .tsf format, analyses series with tsfeatures and catch22, and evaluates univariate and global baselines.

  • Results

    The archive provides 20 publicly available datasets, feature analyses, and baseline forecasts and evaluation results across eight error metrics; feature analysis identifies distinct dataset characteristics.

  • Takeaways & Limitations

    The archive enables researchers to benchmark new global forecasting algorithms against common baselines over related time series from varied domains.

  • Takeaways & Limitations

    Some baseline evaluations were omitted because their computational cost was infeasible in the experimental environment, and error measures are not directly comparable across datasets with different horizons.

Abstract

from arXiv · show

Many businesses and industries nowadays rely on large quantities of time series data making time series forecasting an important research area. Global forecasting models that are trained across sets of time series have shown a huge potential in providing accurate forecasts compared with the traditional univariate forecasting models that work on isolated series. However, there are currently no comprehensive time series archives for forecasting that contain datasets of time series from similar sources available for the research community to evaluate the performance of new global forecasting algorithms over a wide variety of datasets. In this paper, we present such a comprehensive time series forecasting archive containing 20 publicly available time series datasets from varied domains, with different characteristics in terms of frequency, series lengths, and inclusion of missing values. We also characterise the datasets, and identify similarities and differences among them, by conducting a feature analysis. Furthermore, we present the performance of a set of standard baseline forecasting methods over all datasets across eight error metrics, for the benefit of researchers using the archive to benchmark their forecasting algorithms.

1. Introduction

The paper addresses the lack of comprehensive forecasting archives containing related series for evaluating global forecasting models. It introduces an archive, storage format, feature analysis, and baseline evaluations to support benchmarking.

  • Motivation: Global forecasting models train a single model across all series requiring forecasts and can learn cross-series information during training.The paper contrasts these models with local models and relates them to successful competition approaches.
  • Motivation: Existing forecasting archives mostly treat individual, unrelated series as separate datasets, limiting evaluation of global models designed for related series.The paper identifies a gap in comprehensive archives focused on collections of related time series.
  • Contributions: The archive contains 20 publicly available datasets of related series, with 50 dataset variations spanning frequencies, missing-value settings, domains, and series lengths.It includes both real-world and competition datasets.
  • Contributions: The paper introduces a flexible .tsf storage format that records dataset metadata and series-specific information without redundancy.The format is based on Weka ARFF and addresses shortcomings observed in sktime’s .ts format.
  • Contributions: Feature analysis using tsfeatures and catch22 characterises similarities and differences across series, with extracted features released publicly.The analysis is intended for further research use.
  • Contributions: Baseline forecasts from traditional univariate and global models are evaluated across all datasets using eight error metrics, with forecasts and results publicly available.Implementations for data loading, feature calculation, and evaluation are also released.

2. Datasets

The archive organises forecasting data from varied sources and domains, expanding 26 primary datasets into 50 derived datasets. It documents dataset characteristics and supports multiple frequencies and missing-value treatments.

  • Archive scope: The archive contains 20 time series datasets plus six single, very long time series included for machine-learning benchmarking.The paper notes that systematic benchmark data in this space are otherwise lacking.
  • Archive scope: 50 datasets are derived from 26 primary datasets, including separate frequency versions and paired versions with and without missing values.Seven datasets contain missing values, and imputed versions replace them using an appropriate technique.
  • Data sources: The primary datasets come from competitions, research sources, R packages, Kaggle, Johns Hopkins, and domain-specific platforms.The archive draws on multiple source types rather than a single repository.
  • Data sources: The datasets mainly cover tourism, banking, web, energy, sales, economics, transportation, health, and nature.The archive includes varied domain categories.
  • Documentation: The paper introduces key characteristics and prior-use citations for all datasets, with further details organised after the data-format description.Table 1 summarises the primary datasets.

2.1. Data Format

The paper’s .tsf format extends ARFF to represent time series alongside dataset and series attributes. It stores metadata flexibly and supports variable-length series and missing values.

  • Format design: Each .tsf file can store dataset-level attributes, series-level attributes, and data-point-level values or timestamps.Examples include forecasting horizon, missingness, series names, hierarchy position, values, and timestamps.
  • Format design: The format uses tags such as @frequency, @horizon, @missing, and @equallength to describe dataset metadata.Users may define attributes freely, and readers do not require these values to follow a fixed definition.
  • Data representation: After the @data tag, attribute values are colon-separated and each time series is appended as a comma-separated variable-length vector.Variable-length vectors extend the original ARFF representation.
  • Data representation: The example file format is illustrated for the NN5 daily dataset.The figure is an example of the archive’s file representation rather than a forecasting result.
  • Implementation: Missing values are encoded with the “?” symbol, and loaders for R and Python are publicly available.The implementation supports direct use of datasets stored in the format.

2.2. Time Series Datasets

The archive includes competition datasets spanning multiple frequencies, domains, and series counts, with documented prior research using them. The M1, M3, and M4 datasets provide progressively larger benchmark collections.

  • M1 Dataset: The M1 dataset contains 1001 series across yearly, quarterly, and monthly frequencies and seven domains.Its domains include macro, micro, industry, and demographic categories.
  • Prior use: The paper documents prior research using the M1, M3, and M4 datasets, including neural, statistical, hybrid, and feature-based forecasting methods.The cited work demonstrates their use across multiple forecasting approaches.
  • M3 Dataset: The M3 dataset contains 3003 series across yearly, quarterly, and monthly frequencies and six domains.Its domains include demographic, micro, macro, industry, finance, and other.
  • Prior use: Research using M3 includes theta, recurrent-neural-network, evaluation, metric, matrix-factorisation, and complex-seasonality studies.The cited studies span forecasting methods and evaluation research.
  • M4 Dataset: The M4 dataset contains 100,000 series across six frequencies: yearly, quarterly, monthly, weekly, daily, and hourly.Its domains are demographic, micro, macro, industry, finance, and other, and it includes a subset of ForeDeCk series.
  • Prior use: Research using M4 includes hybrid exponential-smoothing and recurrent-neural-network methods, feature-based averaging, and studies of competition-data representativeness.Additional cited work covers probability forecasts and weekly forecasting baselines.

2.2.4. Tourism Dataset

The Tourism dataset contains 1,311 tourism-related time series across yearly, quarterly, and monthly frequencies. It has been used in research on tourism forecasting, structural components, decomposition, and model selection.

  • 1,311 tourism-related time series are included at yearly, quarterly, and monthly frequencies.
  • The dataset is also used in research on recurrent neural networks for time-series forecasting.
  • Research using the dataset includes tourism-demand meta-analysis and forecasting with time-varying parameter structural time-series models.
  • Other studies apply multi-frequency structural components, STL decomposition, and novel model-selection approaches to tourism demand forecasting.

2.2.6. CIF 2016 Dataset

The CIF 2016 competition dataset contains 72 monthly series from banking and artificial sources, with forecast horizons of 6 or 12. It has supported research using recurrent, clustering, bootstrap, exponential-smoothing, and fuzzy approaches.

  • 72 monthly time series comprise the CIF 2016 dataset, including 24 banking series and 48 artificially generated series.
  • 57 series use a forecast horizon of 12, while the remaining 15 use a horizon of 6.
  • Research applies recurrent neural networks and clustering approaches to groups of CIF time series.
  • Other studies combine bootstrap aggregation, clustering, and exponential smoothing, or use fuzzy models with automatic calibration.
  • Time-series clustering using numerical and fuzzy representations has also been studied on this dataset.

2.2.9. Electricity Dataset

The archive includes electricity datasets spanning hourly client consumption and half-hourly household energy use, with both original and aggregated or imputed versions. These datasets support research on recurrent, LSTM, and deep-learning forecasting methods.

  • The client-consumption dataset contains 321 hourly electricity series from 2012 to 2014, measured in kilowatts.
  • The repository also provides a weekly aggregated version of the client electricity-consumption dataset.
  • 5,560 half-hourly household electricity series cover London energy consumption from November 2011 to February 2014.
  • The London household dataset preserves both its original missing-value version and a version with missing values replaced using last-observation carry-forward.
  • Research uses deep recurrent neural networks, scalable LSTMs, and RNN-LSTM approaches for electricity-load forecasting.

2.2.14. FRED-MD Dataset

The FRED-MD dataset contains 107 monthly macroeconomic series beginning in 1959. The series are differenced and log-transformed, and the dataset has been used to study forecasting groups of time series.

  • 107 monthly macroeconomic time series begin on 01/01/1959 and come from the Federal Reserve Bank’s FRED-MD database.
  • The series are differenced and log-transformed as suggested in the literature.
  • The dataset is used in research on locality and globality when forecasting groups of time series.

2.3. Single Long Time Series Datasets

The archive includes six single very long time series from domains including sunspots, river flow, births, electricity demand, and renewable power. These datasets vary substantially in sampling frequency, duration, and scale, with imputed alternatives provided for missing sunspot values.

  • Dataset scope: Six additional benchmark datasets contain single very long time series, complementing the archive’s primary related-series datasets.These include sunspot numbers, Saugeen River flow, US births, Victorian electricity demand, and Australian solar and wind power production.
  • Sunspot numbers: The sunspot dataset represents daily observations from 08/01/1818 to 31/05/2020 and includes an LOCF-imputed version because the original contains missing values.
  • River flow: The Saugeen River dataset contains 23,741 daily mean-flow observations from 01/01/1915 to 31/12/1979.
  • US births: The US births dataset contains 7,305 daily observations from 01/01/1969 to 31/12/1988.
  • Energy series: The Victorian electricity-demand dataset contains 17,520 half-hourly observations from 2014, while Australian solar and wind power series are recorded every 4 seconds.The solar and wind series contain 7,397,222 and 7,397,147 observations, respectively.

3. Feature Analysis

The feature analysis uses extracted time-series characteristics and PCA to compare 20 datasets in a two-dimensional space. It reveals systematic differences associated with frequency, domain, and dataset composition.

  • Feature extraction: The analysis extracts 42 tsfeatures features and examines five features—ACF1, trend, entropy, seasonal strength, and lambda—in detail.The five-dimensional feature space is reduced to two dimensions using PCA for cross-dataset visualisation.
  • Feature extraction: PCA components are constructed from random samples of at most 300 series per large dataset, then all series are mapped into the resulting feature space.This prevents datasets with many series from dominating the components.
  • Dataset characteristics: The M competition datasets concentrate toward high trend and ACF1, whereas car parts, hospital, and Kaggle web traffic concentrate toward higher entropy.Intermittent series are identified as a major reason for the higher entropy in Kaggle web traffic and car parts.
  • Dataset characteristics: Monthly datasets generally have higher seasonal strength, while weekly, daily, and hourly datasets generally have lower seasonal strength except NN5 weekly and NN5 daily.
  • Dataset similarities: M3, M4, and tourism yearly datasets show similar feature-space shapes and density populations, while many daily and weekly datasets differ substantially.M3 and M4 monthly datasets, and electricity and traffic hourly datasets, also show similar patterns.

4. Baseline Forecasting Models

The archive benchmarks six traditional univariate methods and a globally trained pooled regression model using dataset-appropriate horizons and error metrics. Baseline performance varies by frequency and dataset characteristics, with computational constraints limiting some evaluations.

  • Models: Seven baselines are evaluated: ETS, ARIMA, SES, Theta, TBATS, DHR-ARIMA, and globally trained pooled regression (PR).
  • Evaluation design: Evaluation uses fixed-origin forecasts with competition horizons or dataset-specific horizons, including 12 months for monthly data, 8 weeks for weekly data, and 30 days for daily data.TBATS and DHR-ARIMA replace ETS and ARIMA for selected multi-seasonal high-frequency datasets.
  • Scope boundary: The evaluations exclude London smart meters, wind farms, solar power, and wind power for both model families, plus Kaggle web traffic daily for PR, because computation was infeasible.
  • Evaluation design: Performance is assessed with MASE, sMAPE, modified sMAPE, MAE, and RMSE, reporting mean and median values across series.Modified sMAPE is used when zeros can cause division by zero in the original metric.
  • Results: SES performs worst and Theta second-worst across error metrics, while ETS and ARIMA outperform both on yearly, monthly, quarterly, and daily datasets.
  • Results: TBATS generally outperforms DHR-ARIMA on high-frequency datasets, and both outperform SES and Theta.DHR-ARIMA used a maximum Fourier-term order of k = 1, which could be tuned for better results.
  • Results: PR performance is mixed overall but improves with more lagged values and is generally better than traditional univariate models on high-frequency and intermittent datasets.Increasing lags also increases memory and computational requirements.

5. Conclusion

The paper presents a 20-dataset archive focused mainly on related time series, characterises those datasets through feature analysis, and evaluates seven baseline forecasting models across eight error metrics.

  • 20 publicly available datasets from varied domains and frequencies form the archive, with six additional datasets containing single but very long series.The archive’s main focus is sets of time series, while the six single-series datasets broaden its coverage to very long observations.
  • Feature analysis using tsfeatures and catch22 identifies similarities and differences among series in the archive.The features were extracted from each series and made available for further research use.
  • Seven baseline models, including six traditional univariate models and the global model PR, were evaluated across eight error metrics.The evaluation is intended to let researchers benchmark their forecasting algorithms directly against these baselines.
Loading 2105.06643v1…