PaperlayerView exact revision on arXiv

Source-linked AI summary

Exploring Progress in Multivariate Time Series Forecasting: Comprehensive Benchmarking and Heterogeneity Analysis

Zezhi Shao, Fei Wang, Yongjun Xu, Wei Wei, Chengqing Yu, Zhao Zhang, Di Yao, Tao Sun, Guangyin Jin, Xin Cao, Gao Cong, Christian S. Jensen, Xueqi Cheng

arXiv:2310.06119v2cs.LGcs.AI

TL;DR

MTS forecasting studies produce inconsistent or seemingly contradictory findings, complicating reliable assessment of competing approaches. The paper introduces BasicTS+, a fair and reproducible benchmark, and classifies dataset heterogeneity across temporal and spatial characteristics. Its evaluation shows that prior conclusions apply only to certain data types, while current architectural efforts are effective mainly when data exhibit strong patterns.

  • Problem

    MTS forecasting studies report inconsistent or seemingly contradictory performance findings, limiting reliable understanding of competing approaches and technical directions.

  • Method

    The paper introduces BasicTS+ with a unified training pipeline and evaluation settings, classifies temporal and spatial dataset heterogeneity, and evaluates forecasting solutions across heterogeneous datasets.

  • Results

    Prior conclusions hold only for certain data types, including differences in when advanced networks, linear models, and spatial-dependency methods are effective.

  • Takeaways & Limitations

    Selecting MTS forecasting solutions should account for dataset temporal and spatial characteristics rather than generalizing findings across all datasets.

  • Takeaways & Limitations

    Current architectural efforts appear effective mainly when time-series data exhibit strong patterns, while real-world data can show distribution drift and unpredictable changes.

Abstract

from arXiv · show

Multivariate Time Series (MTS) analysis is crucial to understanding and managing complex systems, such as traffic and energy systems, and a variety of approaches to MTS forecasting have been proposed recently. However, we often observe inconsistent or seemingly contradictory performance findings across different studies. This hinders our understanding of the merits of different approaches and slows down progress. We address the need for means of assessing MTS forecasting proposals reliably and fairly, in turn enabling better exploitation of MTS as seen in different applications. Specifically, we first propose BasicTS+, a benchmark designed to enable fair, comprehensive, and reproducible comparison of MTS forecasting solutions. BasicTS+ establishes a unified training pipeline and reasonable settings, enabling an unbiased evaluation. Second, we identify the heterogeneity across different MTS as an important consideration and enable classification of MTS based on their temporal and spatial characteristics. Disregarding this heterogeneity is a prime reason for difficulties in selecting the most promising technical directions. Third, we apply BasicTS+ along with rich datasets to assess the capabilities of more than 45 MTS forecasting solutions. This provides readers with an overall picture of the cutting-edge research on MTS forecasting. The code can be accessed at https://github.com/GestaltCogTeam/BasicTS.

I. INTRODUCTION

MTS forecasting studies face incomparable or contradictory findings, motivating a fair benchmark and analysis of dataset heterogeneity. BasicTS+ evaluates popular solutions comprehensively while examining when conclusions hold across temporal and spatial data characteristics.

  • MTS analysis must model complex temporal dynamics and spatial dependencies that can substantially affect prediction accuracy.
  • Reported experimental findings are often incomparable or inconsistent, creating uncertainty about which technical directions to pursue.Some studies reported DCRNN and GWNet performance up to 33% lower than reproduced results, while LTSF studies often used only normalized-data MAE and MSE.
  • BasicTS+ provides a unified training pipeline and reasonable evaluation settings for fair, comprehensive, and reproducible comparison.It evaluates over 45 popular MTS forecasting solutions on 20 commonly used datasets.
  • The study classifies MTS heterogeneity by temporal patterns and spatial sample indistinguishability to support selecting technical approaches.Temporal categories include stable, drifting, and unclear patterns; spatial categories distinguish datasets with and without significant indistinguishability.
  • Previous conclusions apply only to certain data types, so blindly generalizing them can lead to misguided inferences.For example, basic neural networks outperform advanced ones only on datasets without stable temporal patterns, while spatial-dependency methods are effective only under particular spatial characteristics.
  • The benchmark and heterogeneous datasets support extensive comparison of model performance and efficiency, helping researchers choose solutions or evaluation datasets.

II. RELATED WORK

Related work spans long-term and spatial-temporal forecasting, but recent studies question whether increasingly complex architectures consistently improve MTS prediction. Benchmark results motivate closer attention to data patterns, efficiency, and spatial dependencies.

  • Long-term Time Series Forecasting: LTSF research targets long-range temporal dependencies, progressing from statistical and machine-learning methods toward TCNs, LSTMs, and Transformers.Traditional methods often struggle with high non-linearity and rely heavily on stationarity assumptions.
  • Long-term Time Series Forecasting: LTSF-Linear challenged Transformer effectiveness, while subsequent studies reported advanced neural networks outperforming the linear model.
  • Spatial-Temporal Forecasting: STF methods model temporal dynamics together with dependencies among multiple time series, using CNN-RNN combinations, graph convolutions, attention, and neural architecture search.
  • Spatial-Temporal Forecasting: Graph-based STGNNs typically incur O(N^2L) to O(N^2L^2) complexity, while STNorm and STID achieve similar prediction performance with higher efficiency.Here, N is the number of time series and L is the time-series length.

C. MTS Forecasting Benchmarking

Prior MTS benchmarks cover limited tasks or datasets and do not fully resolve inconsistent findings. BasicTS+ broadens benchmarking across STF and LTSF while linking fair evaluation to dataset heterogeneity and forecasting-task definitions.

  • Existing benchmarks address STF or LTSF tasks but have limitations in coverage, motivating a broader MTS forecasting benchmark.
  • The study distinguishes itself from BasicTS by focusing on reliable evaluation, dataset heterogeneity, and inconsistent findings rather than solely proposing a software benchmark.BasicTS+ was refactored to support both STF and LTSF and includes more extensible features.
  • Table I compares GWNet and DCRNN results reported in highly cited papers with BasicTS+ performance using a relative gap defined as (x − y)/x · 100%.Pink marks the worst reported performance and green marks BasicTS+ performance.
  • MTS forecasting uses historical X ∈ R^(T_h×N) to predict the nearest future Y ∈ R^(T_f×N).An MTS contains multiple time-dependent variables represented as a matrix with T time steps and N variables.
  • BasicTS+ is designed for fair, comprehensive, and reproducible evaluation of both STF and LTSF forecasting solutions.

A. Unified Training Pipeline

BasicTS+ uses a unified training pipeline to reduce spurious performance variation and enable fairer comparisons of forecasting models. It standardizes data processing, training, and evaluation components.

  • Motivation: Different papers report considerable performance variation for the same forecasting solutions, even under apparently identical experimental settings.
  • Sources of variation: A codebase analysis identifies data processing, training configurations, and evaluation implementation as primary sources of spurious performance differences.
  • Unified pipeline: BasicTS+ introduces a unified training pipeline with standardized dataloader, runner, and evaluation components.
  • Data processing: The default dataloader uses z-score normalization and adds temporal features such as time-of-day and day-of-week attributes.
  • Training configuration: The runner controls training, validation, and testing while supporting masked MAE, curriculum learning, and gradient clipping.
  • Evaluation: The evaluation component standardizes implementations of MAE, RMSE, MAPE, WAPE, MSE, and their masked variants.

B. Evaluation Settings

The paper evaluates forecasting on re-normalized data with multiple metrics and analyzes dataset heterogeneity across temporal and spatial characteristics. These settings expose differences that normalized-data evaluation or pooled datasets can obscure.

  • Evaluation settings: MAE and MSE on normalized data can appear deceptively low and are less intuitive because absolute errors depend on data range.
  • Evaluation settings: Re-normalized evaluation with MAE, RMSE, MAPE, and WAPE makes prediction performance less satisfactory but more interpretable across data scales.
  • Implications: Ignoring dataset heterogeneity can produce biased conclusions about technical directions, including claims that Transformer architectures are ineffective.
  • Temporal aspect: Temporal heterogeneity is organized into stable patterns, significant distribution drift, and unclear patterns, which the paper argues are progressively less predictable.
  • Temporal aspect: The temporal visualizations use t-SNE to reduce datasets to two dimensions and kernel density estimation to compare training and testing distributions.
  • Temporal aspect: PEMS03 exhibits fixed-period traffic periodicity, whereas ETTh2 shows non-fixed cycles and shifting means that indicate distribution drift.
  • Model selection: The paper argues that advanced models fit complex stable patterns, while basic models generally perform better under distribution drift or unclear patterns.

B. Spatial Aspect

The paper operationalizes spatial indistinguishability through quantitative metrics and uses it to guide when spatial dependencies should be modeled in MTS forecasting.

  • Spatial indistinguishability: Spatial indistinguishability occurs when similar historical samples have different future data, which simple regression models cannot distinguish.Samples use sliding windows with historical length T_p and future length T_f.
  • Quantitative metrics: The proposed metrics r1 and r2 quantify indistinguishability from complementary perspectives.r1 measures the ratio among all samples, while r2 measures the ratio among samples with similar historical data.
  • Dataset heterogeneity: ETT, Electricity, ExchangeRate, and Weather have very low r1 and r2, whereas METR-LA, PEMS-BAY, PEMS04, and PEMS08 have substantially higher values.These groups share the same data format but are rarely combined in experimental studies.
  • Implications: Spatial modeling can improve performance on datasets with significant spatial indistinguishability but may degrade performance when indistinguishability is low.The paper treats spatial indistinguishability as an indicator for deciding whether spatial dependencies should be modeled.
  • Evaluation: The study evaluates this spatial-indistinguishability hypothesis in Section VI-C.The hypothesis concerns the benefits or risks of modeling spatial dependencies under different dataset conditions.

VI. EXPERIMENTS

The experiments assess hypotheses about technical approaches and use BasicTS+ to compare popular MTS forecasting models and characterize research progress.

  • Experimental objectives: The experiments evaluate hypotheses about temporal and spatial technical approaches while comparing popular MTS forecasting models.The analysis is based on BasicTS+ and is intended to provide insight into progress in MTS forecasting.

A. Experimental Setup

The experimental setup uses heterogeneous MTS datasets, broad baseline coverage, and task-specific forecasting settings to compare forecasting solutions.

  • Datasets: The experiments use 14 datasets spanning traffic, energy, weather, exchange-rate, and general time-series forecasting.The datasets include METR-LA, PEMS variants, ETT variants, Electricity, Weather, ExchangeRate, and M4.
  • Baselines: The baseline suite includes both LTSF and STF models with publicly available official code.Models are categorized according to their technical approaches.
  • STF baselines: STF baselines cover prior-graph, latent-graph, and non-graph approaches.The listed graph-based methods either use predefined graphs or jointly learn graph structures with STGNNs.
  • LTSF baselines: LTSF baselines include Transformer variants and simple linear models.Informer, Autoformer, FEDformer, Triformer, Pyraformer, Crossformer, and PatchTST represent Transformer-based methods, while Linear, DLinear, and NLinear represent linear methods.
  • Classic baselines: The comparison also includes LGBM, DeepAR, and NBeats as classic forecasting models used in industrial applications.LGBM is a gradient-boosting framework, while DeepAR and NBeats are deep-learning solutions.
  • Implementation scope: Additional baselines, including STGODE, NHiTS, and TimesNet, are provided in the repository but omitted from the main presentation due to space limitations.The paper does not cover all BasicTS+ baselines in the main text.
  • Forecasting settings: The STF task uses 12 historical and future steps, while LTSF uses a 336-step future horizon and historical lengths of 96, 192, 336, and 720.The reported LTSF results use the best prediction performance across the tested historical lengths.

4) Hyperparameter tuning:

The experiments compare advanced and basic neural networks across heterogeneous datasets and examine spatial modeling choices under different spatial-indistinguishability conditions.

  • Advanced Neural Networks vs. Basic Neural Networks: Advanced models generally outperform basic models on datasets with clear and stable patterns.The comparison includes Transformer-based advanced models and linear basic models.
  • Advanced Neural Networks vs. Basic Neural Networks: Basic models consistently outperform advanced models on datasets with distribution drift or unclear patterns.The experiments compare these model classes using MAE, RMSE, and WAPE.
  • Hyperparameter tuning:: On PEMS08, FEDformer’s training, validation, and testing MAEs decrease with more epochs, whereas DLinear’s MAEs do not decrease, indicating under-fitting.The training, validation, and testing errors start from similar values for FEDformer.
  • Hyperparameter tuning:: On ETTh2, FEDformer’s training MAE decreases while validation and testing MAEs increase after two epochs, indicating serious over-fitting.The training and evaluation curves diverge early during training.
  • Advanced Neural Networks vs. Basic Neural Networks: Models with less inductive bias usually perform better when no explicit pattern exists.The paper gives MLP-based models and vanilla Transformers as examples.
  • C. Delving into Spatial Dependencies: Trainable spatial identity embeddings and GCNs improve performance on datasets with significant spatial indistinguishability.The comparison uses STID and AGCRN variants with their spatial modeling components removed or neutralized.
  • C. Delving into Spatial Dependencies: Adding spatial modeling components degrades performance on datasets with low spatial indistinguishability.The result suggests that modeling spatial dependencies is not always necessary.

D. Performance and Efficiency Benchmarking

The benchmarking framework links MTS model selection and dataset selection to temporal and spatial characteristics, then evaluates forecasting solutions across task-aligned and compound datasets.

  • Benchmark design: The study combines detailed LTSF and STF evaluations with comparisons on the complex M4 competition dataset.Representative STF, LTSF, and classic time-series solutions are included in the M4 comparison.
  • Model and dataset selection: Temporal patterns should be examined first when selecting or designing an MTS forecasting solution.Datasets with drift or unclear patterns favor linear layers, MLPs, or vanilla Transformers, while stable patterns favor TCNs, RNNs, or Transformer architectures.
  • Model and dataset selection: Spatial dependency modeling is recommended when datasets exhibit high sample indistinguishability across series.Suggested approaches include graph convolution, spatial-temporal normalization, and spatial identity attaching; STID and Linear are recommended baselines.
  • Evaluation dataset selection: STF evaluation requires datasets with significant spatial dependency, whereas LTSF evaluation should include datasets with and without clear, stable patterns.Dataset characteristics should align with the task objective being tested.
  • Evaluation dataset selection: Compound datasets such as M4 are more suitable for practical ranking because they combine multiple challenges across time series.A solution designed for one task type may not outperform others on such datasets.

3) Experimental Results:

The experiments show that model complexity does not consistently determine MTS forecasting performance: simple or older methods remain competitive across LTSF, STF, and compound industrial data.

  • LTSF results: Advanced Transformers and basic linear models perform comparably on LTSF datasets without clear and stable patterns.For these tasks, training strategies and feature engineering for distribution drift or ambiguous patterns are identified as more important challenges than increasing architectural complexity.
  • LTSF results: Many recent LTSF solutions struggle to outperform Informer on datasets with clear and stable patterns.The result raises concerns that existing architectures may be over-fitted to commonly used datasets such as ETT, Electricity, Weather, and ExchangeRate.
  • STF results: Prior-graph-based STF methods generally outperform latent-graph-based and non-graph-based methods.Only MTGNN and STEP learn effective graph structures without significantly degrading prediction performance.
  • STF results: 6% higher MAE is achieved by D2STGNN than Graph WaveNet on METR-LA despite D2STGNN being published three years later.Graph WaveNet and MTGNN also significantly outperform several newer solutions, including StemGNN and GTS.
  • M4 results: PatchTST and STID, despite being superior in their respective LTSF and STF tables, perform worse than industrially used solutions on M4.The M4 evaluation covers Yearly, Quarterly, Monthly, and Others subsets with weighted averages.
  • Limitations and future directions: Current deep-learning efforts appear effective mainly when MTS data exhibits strong patterns, while real-world data also faces distribution drift, insufficient volume, and low quality.The paper identifies these factors as bottlenecks and calls for research on distribution shifts, low-quality data, and zero- or few-shot learning.

APPENDIX

The appendix documents the benchmark datasets and baseline overview, spanning traffic, electricity, weather, exchange-rate, and large-scale competition time series.

  • Dataset descriptions: METR-LA and PEMS-BAY contain traffic-speed data recorded every five minutes and include sensor graphs representing spatial dependencies.These datasets support spatial-temporal forecasting evaluation.
  • Dataset descriptions: PEMS03, PEMS04, PEMS07, and PEMS08 contain traffic-flow data recorded every five minutes with sensor graphs.The graphs indicate dependencies between sensors.
  • Dataset descriptions: ETTh1, ETTh2, ETTm1, and ETTm2 record electricity-transformer temperatures at hourly or 15-minute sampling intervals.The dataset names distinguish transformers and sampling frequencies.
  • Reference tables: Tables VIII and IX provide dataset statistics and a brief overview of the study’s baselines.The appendix also identifies the dataset and baseline summary tables.
  • Dataset descriptions: Electricity records hourly electricity consumption in kWh for 321 clients from 2012 to 2014.Weather records 21 meteorological indicators every 10 minutes during 2020, while ExchangeRate contains daily exchange rates for eight countries.
  • Dataset descriptions: M4 contains 100,000 non-aligned time series collected at yearly, quarterly, monthly, weekly, daily, and hourly frequencies.The series have different lengths and start and end times.
Loading 2310.06119v2…