PaperlayerView exact revision on arXiv

Source-linked AI summary

A Critical Audit of Spatiotemporal Forecasting Benchmark Datasets and Baselines

Kenneth Martin, Simon Heilig, Asja Fischer, Michel F. C. Haddad, Adam M. Sykulski, Moshe Eliasof

arXiv:2608.20980v1cs.LGstat.ML

TL;DR

Widely used spatiotemporal forecasting benchmarks may favor spatially uninformed linear models, challenging their reliability for comparing GNNs. This paper analyzes temporal and spatial correlations, re-evaluates linear baselines, and finds that SARIMA exposes a performance gap while residual-targeted GNNs can improve performance.

  • Problem

    Spatiotemporal forecasting research relies heavily on a limited set of benchmarks where simple baselines can rival or outperform GNNs, raising concerns about their discriminative reliability.

  • Method

    The paper analyzes lagged spatial and temporal correlations across five benchmarks, re-evaluates spatially uninformed baselines, and trains GNNs to predict SARIMA residuals.

  • Results

    Most datasets are dominated by node-wise temporal information; SARIMA reveals a performance gap with state-of-the-art GNNs, while residual-targeted GNNs improve performance.

  • Takeaways & Limitations

    Benchmark comparisons should reduce reliance on these datasets and incorporate more rigorous statistical evaluation of spatial and temporal structure.

  • Takeaways & Limitations

    The correlation metrics capture only linear dependency structures.

Abstract

from arXiv · show

Graph neural networks (GNNs) are routinely employed for short-range forecasting on multivariate time series with a spatial graph structure. Despite the availability of many alternative datasets, method innovations within this domain are predominantly assessed against a rather limited set of benchmark datasets, most notably Chickenpox, PedalMe, WikiMaths, METR-LA, and PEMS-BAY. The evaluation protocols contain baselines spanning from historical averages to classical machine learning approaches. These baselines often show competitive performance compared to GNNs. In the present work, we take a step back and analyse the benchmark datasets via classical time series methods to uncover why spatially-unaware linear models pose a stronger competitor than previously reported, casting further doubt on the discriminative reliability of the aforementioned widely adopted datasets. Our statistical analysis provides a toolset for identifying significant spatial and temporal correlations, while revealing a structural bias introduced by first-order differenced datasets. We therefore recommend reducing the over-reliance on such datasets for method comparison, and instead advocate for more rigorous statistical evaluation. By applying the results of our analysis to a simple hybrid model, we show how our methodology can lead to novel ways of developing GNN models

1 Introduction

This work audits five widely adopted spatiotemporal forecasting benchmarks through statistical correlation analysis and baseline re-evaluation. It identifies shortcomings in differenced-data evaluation and motivates SARIMA-residual GNN training, reaching state-of-the-art performance levels on METR-LA and PEMS-BAY.

  • 1 Introduction: Constant-mean and node-wise ridge baselines showed early evidence that nonlinear models underperformed on Chickenpox, PedalMe, and WikiMaths [Micheli and Tortorella, 2022].On METR-LA and PEMS-BAY, early strong approaches used time and space identifiers [Shao et al., 2022a].
  • 1 Introduction: The study statistically analyzes lagged temporal and spatial correlations across Chickenpox, PedalMe, WikiMaths, METR-LA, and PEMS-BAY.These are identified as the five widely adopted benchmark datasets.
  • 1 Introduction: Evaluation protocols often compare models on differenced rather than raw time series, producing inferior predictive performance across the board.This is presented as a major shortcoming of existing evaluation protocols.
  • 1 Introduction: Training GNNs on SARIMA residuals raises predictive performance to the level of state-of-the-art methods on METR-LA and PEMS-BAY.The residual-based target is motivated by re-evaluating spatially uninformed SARIMA baselines on Chickenpox and WikiMaths.
  • 1 Introduction: A synthetic experiment indicates that linear-model gains partially arise from greater adaptiveness to heterogeneity in node-wise time-series properties.The finding helps motivate spatially uninformed baseline re-evaluation, including SARIMA on Chickenpox and WikiMaths.

2 Benchmark Datasets and Evaluation Protocols

The benchmark datasets exhibit substantially stronger temporal than spatial dependence, with traffic datasets retaining the clearest residual spatial signal after controlling for time. Default differencing can distort these properties, potentially emphasizing noise and undermining evaluation protocols.

  • Correlation Findings: Chickenpox and PedalMe show negative lag-1 temporal correlation consistent with overdifferencing, while PedalMe’s short series make its correlation estimates highly volatile.Both datasets are released in PyTorch Geometric Temporal as differenced series rather than absolute values.
  • Correlation Findings: WikiMaths has a clear weekly cycle, and its default eight-step loader covers this seasonality while partial spatial correlation retains weekly information.The weekly pattern remains after controlling for temporal correlation, indicating residual spatial structure at that cycle.
  • Correlation Findings: Traffic datasets show the strongest residual spatial information, but all benchmarks generally exhibit weaker spatial than temporal correlation.METR-LA and PEMS-BAY have strong, slowly decaying temporal and spatial correlations, while partial spatial correlations remain stable but much smaller after temporal adjustment.
  • Differencing Effects: First-order differencing acts as a high-pass filter that can make models overfit high-frequency noise when predictive signal is concentrated at low frequencies.This mechanism is especially problematic when measurement error and noise are high-frequency concentrated while the true signal is low-frequency concentrated.
  • Evaluation Protocols: Undoing Chickenpox differencing substantially changes its correlation structure, revealing persistent temporal dependence and exposing a shortcoming in protocols trained and tested on differenced data.The analysis uses linear correlation measures, which are considered sufficient here for selecting appropriate baselines and evaluation protocols despite not capturing nonlinear dependencies.

3 Spatially Uninformed Baselines

Spatially unaware temporal baselines perform strongly on these benchmarks, sometimes matching or exceeding state-of-the-art models. A SARIMA-based residual-learning hybrid reaches state-of-the-art on METR-LA and PEMS-BAY.

  • 3 Spatially Uninformed Baselines: The section investigates simple temporal models because benchmark performance is strongly driven by temporal signals, finding that even simple linear models can approach state-of-the-art.The authors therefore propose SARIMA as an improved linear benchmark for smaller datasets and combine it with a spatial GNN on larger traffic datasets.
  • 3.1 Existing Linear Baselines: The study evaluates persistence, historical averages, per-node autoregression, RidgeVAR, and DLinear as spatially unaware temporal baselines on Chickenpox, PedalMe, and WikiMaths.The baselines range from reusing the latest observation or seasonal means to ridge-regularized temporal and cross-node linear models.
  • 3.1 Existing Linear Baselines: AR(H) per node and DLinear are among the best baselines on Chickenpox and PedalMe, while RidgeVAR ranks first on WikiMaths above TDE-GNN.These models are spatially unaware, although RidgeVAR captures linear cross-node dynamics through a fully connected regression.
  • 3.2 SARIMA Baselines: SARIMA outperforms existing state-of-the-art models on WikiMaths, while ARIMA with temporal encodings surpasses TDE-GNN on Chickenpox.SARIMA is a seasonally informed linear model whose order controls the required lags and coefficients; ARIMA-based methods perform poorly only on PedalMe among the reported datasets.
  • 3.3 SARIMA Residual Learning: The SARIMA residual-learning model exceeds previously reported metrics and becomes state-of-the-art on both METR-LA and PEMS-BAY.The method augments a spatial model with SARIMA error features and predictions, enabling a follow-up model to correct temporal forecasts using spatial information.

4 Outlook

The section recommends repeatable benchmark analysis, retaining PedalMe and Chickenpox undifferenced, and using SARIMA as a strong baseline that can motivate SARIMA–GNN hybrids. A synthetic experiment further links temporal heterogeneity to relative GRU-GCN and ARIMA performance while finding spatial-coefficient variance nonsignificant.

  • Benchmark recommendations: The analysis recommends keeping PedalMe and Chickenpox undifferenced for benchmarking.Section 2.1 provides a repeatable procedure for analyzing the spatial and temporal properties of benchmark datasets.
  • Baseline and model development: SARIMA models can provide effective baselines for spatiotemporal graph regression, and integrating SARIMA with GNNs offers a method-development direction.The section presents stronger baselines as a source of inspiration for new methods, exemplified by the SARIMA–GNN integration.
  • Synthetic experiment: The synthetic study investigates whether linear models benefit from avoiding homogeneous spatiotemporal assumptions when nodal time-series properties are heterogeneous.The motivation draws on prior concerns that GNNs may perform poorly under heterophily [Zhu et al., 2020].
  • Synthetic experiment: The experiment simulates an AR(1) graph process and compares GRU-GCN with ARIMA across varying temporal, spatial, and global graph sensitivities.It uses 500 time steps, 10 nodes, and a 90/10 split, with repeated simulations varying coefficient variances and global spatial weighting.
  • Synthetic experiment: Temporal-coefficient variance positively influences the test-MSE gap between GRU-GCN and ARIMA, whereas spatial-coefficient variance is not significant.Changes in the global spatial coefficient and mean nodal variance appear to contain most spatial variation, while the global spatial coefficient is the only parameter positively affecting GNN performance.

5 Conclusion

The paper’s statistical analysis of five widely used spatiotemporal benchmarks finds that most are dominated by node-wise temporal information. It also identifies first-order differencing in the official data loader for Chickenpox and PedalMe as a benchmark-quality issue because it obscures low-frequency signals.

  • Statistical correlation analysis of Chickenpox, PedalMe, WikiMaths, METR-LA, and PEMS-BAY reveals that most datasets are dominated by node-wise temporal information.
  • The official PyTorch Geometric Temporal data loader releases first-order differenced data for Chickenpox and PedalMe.
  • First-order differencing obscures low-frequency signals and weakens the quality of the Chickenpox and PedalMe benchmarks.

A Background details on SARIMA

SARIMA extends ARIMA with seasonal dependence, seasonal differencing, and seasonal autoregressive and moving-average terms. The implementation uses AutoARIMA for scalable order selection while restricting model lags to benchmark input windows when required.

  • Model definition: SARIMA extends ARIMA by adding seasonal autoregressive and moving-average terms for dependence at a fixed period s, such as daily traffic seasonality.Its orders are represented as (p, d, q)(P, D, Q)s, combining nonseasonal and seasonal AR, differencing, and MA components.
  • Implementation details: AutoARIMA implementations in StatsForecast and CuML select SARIMA orders using a minimized criterion and scale fitting to datasets with thousands of observations and hundreds of nodes.For small datasets, restricted searches use AIC, while unrestricted AutoARIMA selects any reasonable order; the distinction materially affected WikiMaths.
  • Lag restriction: The maximum lag of SARIMA is s(P + D) + p + d, so fair window-matched fitting enforces s(P + D) + p + d ≤ H.This accounts for the highest-order seasonal and nonseasonal backshift contributions and can be contrasted with unrestricted training.
  • Implementation details: For large traffic datasets, SARIMA uses the fixed order (4, 1, 4) × (4, 1, 4, 12), selected by a limited grid search, under an unrestricted protocol matching DLinear.Seasonality captures slower traffic-process decay without fitting a parameter for every lag.

B Re-Evaluating Chickenpox

The Chickenpox re-evaluation trains a broad set of spatiotemporal models on officially available data, following Rozemberczki et al. [2021]. Results are reported using train/test MSE and generalization gaps across native and differenced spaces, with comparisons restricted to compatible units.

  • Results: Table 5 reports Chickenpox train/test MSE ± std over n=10 seeds together with the generalization gap, defined as (test − train) / train MSE.The table reports these quantities in both native and differenced space units.
  • Results: MSE is comparable within each evaluation mode but not across modes because raw and loader native units differ; diff-space columns project raw-mode predictions.Raw mode uses z-scored levels, whereas loader mode uses z-scored first differences.

C Synthetic Graph Experiment

The synthetic experiment models a stable graph-based VAR(1) process with heterogeneous temporal and spatial dynamics. It evaluates 2,000 simulated samples across varied graph concentration, spatial sensitivity, and coefficient heterogeneity settings, analyzing the ARIMA–GNN test-MSE gap.

  • Synthetic process: The process is a graph VAR(1), Xt = TXt−1 + ϵt−1, with transition matrix T = TA + λTBA and stability requiring |ρ(T)| < 1.The innovations satisfy ϵt−1 ∼ N(0, σ2IN).
  • Experimental setup: The parameter sweep produces 2,000 samples spanning diverse node-level temporal and spatial heterogeneity.Node-specific coefficients are sampled around 0.5, while adjacency rows follow a Dirichlet distribution with concentration parameter α.
  • Heterogeneity analysis: The ARIMA–GNN test-MSE gap is regressed against heterogeneity measures, with the heatmap focusing on λ and cross-node temporal-coefficient variance.These are identified as the two most significant regression coefficients, and the linear regression has R2 = 0.591.

D Experimental settings

Experiments used distributed CPU resources for large-scale ARIMA searches on METRLA and PEMSBAY, while smaller runs used a laptop GPU. The implementation relied on Python 3.12 and torch-geometric-temporal 0.56.2.

  • D Experimental settings: Large-scale METRLA and PEMSBAY experiments distributed ARIMA grid searches across a 25-CPU cluster, completing in under 30 minutes.
  • D Experimental settings: The large-scale experiments used an Nvidia-A40 46GB GPU, whereas small-scale experiments ran on a home laptop with a GTX 1060 GPU.
  • D Experimental settings: The primary software versions were Python 3.12 and torch-geometric-temporal 0.56.2.
Loading 2608.20980v1…