PaperlayerView exact revision on arXiv

Source-linked AI summary

Spectral Temporal Graph Neural Network for Multivariate Time-series Forecasting

Defu Cao, Yujing Wang, Juanyong Duan, Ce Zhang, Xia Zhu, Conguri Huang, Yunhai Tong, Bixiong Xu, Jing Bai, Jie Tong, Qi Zhang

arXiv:2103.07719v1cs.LGcs.AI

TL;DR

Multivariate forecasting requires jointly modeling temporal dependencies within series and correlations across series, while existing approaches often use time-domain temporal modeling and predefined relational priors. StemGNN combines GFT and DFT in the spectral domain and automatically learns inter-series correlations; across nine public benchmarks, it improves average MAE by 8.1% and RMSE by 13.3% over the best baselines.

  • Problem

    Multivariate forecasting must jointly model intra-series temporal correlations and inter-series correlations, while many existing methods use time-domain temporal modeling and predefined inter-series priors.

  • Method

    StemGNN combines Graph Fourier Transform and Discrete Fourier Transform in an end-to-end spectral-domain framework and learns inter-series correlations automatically from data.

  • Results

    8.1% on MAE and 13.3% on RMSE are StemGNN’s average improvements over the best baseline across nine public benchmarks.

  • Takeaways & Limitations

    StemGNN provides a multivariate forecasting approach that does not require pre-defined topologies and performs consistently across varied forecasting applications.

  • Takeaways & Limitations

    The authors state that current forecasting models remain far from predicting future data absolutely correctly.

Abstract

from arXiv · show

Multivariate time-series forecasting plays a crucial role in many real-world applications. It is a challenging problem as one needs to consider both intra-series temporal correlations and inter-series correlations simultaneously. Recently, there have been multiple works trying to capture both correlations, but most, if not all of them only capture temporal correlations in the time domain and resort to pre-defined priors as inter-series relationships. In this paper, we propose Spectral Temporal Graph Neural Network (StemGNN) to further improve the accuracy of multivariate time-series forecasting. StemGNN captures inter-series correlations and temporal dependencies \textit{jointly} in the \textit{spectral domain}. It combines Graph Fourier Transform (GFT) which models inter-series correlations and Discrete Fourier Transform (DFT) which models temporal dependencies in an end-to-end framework. After passing through GFT and DFT, the spectral representations hold clear patterns and can be predicted effectively by convolution and sequential learning modules. Moreover, StemGNN learns inter-series correlations automatically from the data without using pre-defined priors. We conduct extensive experiments on ten real-world datasets to demonstrate the effectiveness of StemGNN. Code is available at https://github.com/microsoft/StemGNN/

1 Introduction

Multivariate forecasting must jointly model temporal patterns within series and correlations between series. StemGNN addresses this challenge in the spectral domain, automatically learning dependencies and achieving strong benchmark performance.

  • Accurate multivariate forecasting requires jointly modeling intra-series temporal patterns and inter-series correlations.
  • StemGNN combines GFT and DFT to represent structural and temporal dependencies jointly in the spectral domain.GFT transforms structural multivariate inputs, while DFT transforms each univariate series into the frequency domain.
  • StemGNN jointly represents intra-series and inter-series correlations in the spectral domain, combining GFT, DFT, and deep neural networks.The paper identifies this as its first such integrated spectral-domain representation and reports supporting ablation studies.
  • StemGNN learns inter-series dependency graphs automatically from data instead of relying on pre-defined topologies.The learned graphs are reported to be interpretable and to outperform human-defined graph structures in experiments.
  • 8.1% on MAE and 13.3% on RMSE are StemGNN’s average improvements over the best baseline across nine public benchmarks.The introduction also reports state-of-the-art performance on those benchmarks and a COVID-19 case study.

2 Related Work

Related work spans univariate and multivariate forecasting, with methods using recurrent, convolutional, state-space, factorization, and graph-based approaches. Graph neural networks explicitly model inter-series correlations but commonly depend on predefined topologies.

  • Time-series forecasting research is divided into univariate and multivariate techniques.Univariate methods analyze each series separately, whereas multivariate methods consider multiple series together.
  • Univariate methods include LSTM-based forecasting, DFT-enhanced frequency modeling, and deep architectures using basis expansion.The cited examples are FC-LSTM, SMF, and N-BEATS.
  • Multivariate methods use architectures such as dilated temporal convolutions, CNN-RNN combinations, state-space models, and global-local factorization.Examples include TCN, LSTNet, DeepState, and DeepGLO.
  • Figure 1 presents the overall architecture of the proposed Spectral Temporal Graph Neural Network.
  • Graph neural network approaches explicitly capture correlations between time series through graph-based spatiotemporal modeling.DCRNN and ST-GCN combine graph operations with temporal modules for traffic forecasting.

3 Problem Definition

The paper formulates multivariate forecasting on a temporal graph whose nodes are time series and whose weighted edges represent inter-series relationships. Given K historical timestamps, the task is to predict the next H timestamps.

  • A multivariate temporal graph is defined as G = (X, W), with time series as nodes and W as the adjacency matrix.An edge weight wij > 0 indicates a connection between nodes i and j, with wij representing its strength.
  • The input X ∈ R^N×T contains N time series observed across T timestamps, while X_t contains all node values at timestamp t.
  • Given observations from the previous K timestamps, the model predicts node values for the next H timestamps.The forecasting function uses parameters Φ and a graph structure G, which may be provided as a prior or inferred automatically.

4 Spectral Temporal Graph Neural Network

StemGNN learns latent inter-series correlations and models structural and temporal dependencies jointly in the spectral domain. Its architecture combines automatically inferred graphs, GFT, DFT-based sequential processing, graph convolution, residual blocks, and forecasting/backcasting outputs.

  • Latent Correlation Layer: StemGNN infers graph structure and adjacency weights automatically from multivariate input data using a latent correlation layer.A GRU summarizes each series, and self-attention produces the adjacency weight matrix W.
  • Outputs and Training: Forecasting branches predict future values, while backcasting branches reconstruct historical inputs and contribute a combined forecasting-backcasting loss.Inference uses a rolling strategy to generate multi-step forecasts.
  • StemGNN Block: StemGNN blocks combine spectral graph convolution with Spe-Seq processing to model structural and temporal dependencies jointly.Multiple blocks use residual connections, and the model applies GFT and Spe-Seq processing across input channels before IGFT.
  • Spectral Sequential Cell: The Spe-Seq Cell applies DFT, one-dimensional convolution, GLU, and IDFT to learn temporal patterns after graph transformation.Real and imaginary frequency components are processed in parallel with separate parameters.
  • Spectral Graph Convolution: GFT transforms graph signals into a spectral representation, where graph convolution filters structural information before inverse transformation.The graph Fourier basis is formed from eigenvectors of the normalized graph Laplacian, with complexity O(N^3).

5 Experiments

StemGNN is evaluated against state-of-the-art forecasting models on public datasets spanning traffic, energy, and electrocardiogram domains. It achieves strong benchmark performance, learns latent correlations without prior topology, and benefits from its architectural components.

  • Experimental Setup: StemGNN is compared with state-of-the-art models on nine public datasets spanning traffic, energy, and electrocardiogram forecasting.The experiments use dataset-specific train, validation, and test splits and normalized inputs.
  • Forecasting Results: 8.1% lower MAE and 13.3% lower RMSE are reported on average than the best baseline for each dataset.The paper reports these improvements across all evaluated datasets.
  • Forecasting Results: StemGNN establishes a new state-of-the-art on most datasets without requiring an a priori topology.Its learned latent correlations are presented as feasible for datasets lacking predefined structures.
  • Forecasting Results: For traffic forecasting, StemGNN consistently outperforms graph-based models without prior road-network knowledge.The comparison supports data-driven latent correlations over human-defined priors in these experiments.
  • Ablation Study: Ablation results indicate that temporal modeling, DFT, residual connections, backcasting, and GFT each contribute to performance.The study reports that all tested components are indispensable on PEMS07 and evaluates additional datasets in the appendix.
  • Latent Correlation Analysis: The learned traffic correlation matrix assigns different correlation strengths to detectors, indicating that some sensors are more closely related than others.The case study visualizes correlations for six PEMS-BAY detectors.

6 Analysis

StemGNN is analyzed on COVID-19 forecasting, where it outperforms other solutions across multiple horizons and learns interpretable country dependencies. Spectral analysis shows GFT captures shared regional trends and produces smoother series for forecasting.

  • COVID-19 forecasting: StemGNN outperforms other state-of-the-art solutions for COVID-19 forecasting at 7-, 14-, and 28-day horizons.The study uses 25 countries, with 60 days for training and 50 days for testing.
  • COVID-19 forecasting: StemGNN forecasts Singapore’s rapid post-April-14 increase successfully four weeks in advance.The predicted values are described as consistent with ground truth for Brazil, Germany, and Singapore.
  • Learned dependencies: Learned country correlations align with geographic intuition, including strong relationships among the US, Canada, and Mexico and among China, Japan, and Korea.Larger values in the correlation visualization indicate stronger correlations.
  • GFT and DFT analysis: The first two GFT components capture common trends for the world and Asian countries, respectively.These components are compared with the corresponding ground-truth COVID-19 series.
  • GFT and DFT analysis: The transformed result series are smoother and more feasible for forecasting than the series after GFT alone.

7 Conclusion

The paper concludes that StemGNN jointly models inter-series correlations and temporal dependencies in the spectral domain and performs consistently well across multivariate forecasting applications. Future work targets scalability and additional real-world applications.

  • StemGNN jointly models inter-series correlations and temporal dependencies in the spectral domain.
  • StemGNN consistently outperforms existing approaches across varied multivariate time-series forecasting applications.
  • Future work will reduce time complexity for very large graphs and apply StemGNN to additional real-world forecasting and predictive-maintenance scenarios.The paper identifies direct eigenvalue decomposition as prohibitive for very large, high-dimensional graphs.

Broader Impact

The paper presents StemGNN as a general spectral forecasting approach with automatically learned relationships among multivariate signals. It also discusses societal benefits, misuse risks, and the need for responsible AI practices.

  • StemGNN jointly models inter-series correlations and temporal patterns in the spectral domain to improve multivariate-signal representations.
  • Its latent correlation layer learns relationships among multivariate signals automatically in an end-to-end framework.
  • Accurate forecasting can support supply-chain management, daily-life safety decisions, and advance responses to flooding or epidemics.
  • The paper calls for people-centered, responsible AI practices addressing fairness, interpretability, privacy, security, and accountability.It notes that current models remain far from perfectly accurate.

B.1 Datasets

The evaluation covers public datasets spanning traffic, energy, electrocardiogram, and COVID-19 forecasting, with topology available only for traffic data. It compares StemGNN with diverse recurrent, convolutional, graph, spectral, residual, and state-space baselines.

  • Datasets: The benchmark comprises ten public datasets across traffic, energy, electrocardiogram, and COVID-19 domains.Only traffic datasets provide a priori topology.
  • Datasets: Traffic experiments cover flow and speed forecasting on PEMS03, PEMS07, PEMS08, PEMS04, PEMS-BAY, and METR-LA.Traffic data are aggregated into 5-minute intervals from 30-second samples.
  • Datasets: Energy experiments use Solar data from 137 Alabama photovoltaic stations and Electricity data from 370 customers.Solar is sampled every 10 minutes, while Electricity contains hourly consumption series.
  • Datasets: The ECG5000 dataset contains 140 electrocardiograms, each with length 5000.
  • Datasets: COVID-19 experiments use newly confirmed cases from 25 countries over 110 days, split into 60 training days and 50 testing days.
  • Baselines: Baselines include recurrent, convolutional, graph, spectral-frequency, residual, and state-space forecasting models with documented implementation settings.Examples include FC-LSTM, SMF, N-BEATS, LSTNet, DCRNN, STGCN, TCN, DeepState, GraphWaveNet, and DeepGLO.

D Experiment Details

Experiments use normalized inputs, dataset-specific train/validation/test splits, and forecasting horizons tailored to each application. Longer-horizon evaluation on METR-LA and COVID-19 tests multi-step forecasting, where StemGNN reduces time-dependent error accumulation.

  • Experimental setup: Experiments use dataset-specific train, validation, and test splits, with ECG min-max normalization and other inputs Z-Score normalized.All experiments use one NVIDIA GeForce GTX 1080 GPU.
  • Experimental setup: StemGNN uses a 32-dimensional self-attention layer, 64-channel graph convolutions, kernel size 3, batch size 50, and 50 training epochs.The learning rate starts at 0.001 and decays by 0.7 every five epochs.
  • Forecasting settings: Traffic inputs use one hour of history to forecast either the next 15 minutes or the next hour, while COVID-19 forecasting extends up to 28 days.Traffic data are aggregated every five minutes, yielding 288 timestamps per day.
  • Long-horizon evaluation: Long-horizon evaluation on METR-LA and COVID-19 shows excellent multi-step performance and reduced time-dependent error accumulation compared with other solutions.COVID-19 forecasts target infected people one to four weeks ahead to support departmental decision-making.

E.2 More results for ablation study

Ablation studies show that spectral temporal modeling and latent-correlation modeling are central to StemGNN’s performance, while residual and backcasting branches provide additional representation benefits.

  • Correlation modeling: Dynamic latent correlations outperform a static priori topology, because StemGNN builds a topology for each sliding window.The learned topology captures updated interactions between time-series.
  • Spectral temporal modeling: Removing the Spe-Seq Cell produces the worst ablation performance, indicating that temporal dependency is the most important forecasting clue.This setting omits the spectral sequential cell entirely.
  • Spectral temporal modeling: Removing DFT improves over the naive model without the Spe-Seq Cell but underperforms StemGNN by a large margin, demonstrating DFT’s benefit.The ablation models temporal dependencies in the time domain instead.
  • Correlation modeling: Removing GFT causes a significant performance drop, showing the necessity of capturing latent correlations through graph Fourier transform.This variant retains only the Spectral Sequential Cell.
  • Architectural components: Removing residual connections verifies that the second StemGNN block learns supplementary information through the residual pathway.
  • Architectural components: Disabling backcasting reduces performance, showing that the backcasting module enhances time-series representations.

F.1 Efficiency Analysis

StemGNN combines competitive training efficiency with interpretable spectral representations and COVID-19 forecasting behavior. Its learned components produce smoother trends that support forecasting across countries and horizons.

  • Efficiency analysis: StemGNN has O(N^3) time complexity in multivariate dimension N, yet training finishes in reasonable time across all datasets.Its training time is similar to first-order approximate graph convolution STGCN, while performance is significantly improved; training and inference are faster than other baselines.
  • Efficiency analysis: On METR-LA, learning curves plot wall-clock time against validation RMSE, with StemGNN converging to a better RMSE than other SOTAs at comparable training times.
  • COVID-19 case study: COVID-19 latent correlations vary across countries, with each matrix row representing one country’s correlation strength with other countries.
  • COVID-19 case study: Graph Fourier eigenvectors capture global, Asian, and South American trends, while transformed time-series become smoother and easier to forecast.IDFT further improves smoothness and generalization, and exemplar-country forecasts demonstrate StemGNN’s feasibility.
  • COVID-19 case study: COVID-19 comparisons evaluate one-day and one-week forecasts for selected countries, where StemGNN predicts future trends more accurately than two major baselines.GFT and the Spectral Sequential Cell capture major trends more smoothly and predict data changes.
Loading 2103.07719v1…