Pre-training Enhanced Spatial-temporal Graph Neural Network for Multivariate Time Series Forecasting

Zezhi Shao; Zhao Zhang; Fei Wang; Yongjun Xu

Source-linked AI summary

Pre-training Enhanced Spatial-temporal Graph Neural Network for Multivariate Time Series Forecasting

Zezhi Shao, Zhao Zhang, Fei Wang, Yongjun Xu

arXiv:2206.09113v2Jun 18, 2022cs.LG

TL;DR

STGNNs model multivariate time series with short histories, although temporal patterns and inter-variable dependencies may require long-term observations. STEP adds scalable pre-training that extracts segment-level representations from very long histories and uses them to enhance STGNNs. Experiments on three real-world datasets show significant downstream improvements, with TSFormer capturing temporal patterns.

Problem
STGNNs often use short historical windows, limiting access to long-term temporal patterns and reliable information for modeling dependencies between time series.
Method
STEP pre-trains TSFormer with Transformer blocks and masked autoencoding on very long histories, then uses segment representations for contextual STGNN inputs and graph learning.
Results
The framework significantly enhances downstream STGNNs on three real-world datasets, and TSFormer captures temporal patterns.
Takeaways & Limitations
STEP provides stable performance gains for Graph WaveNet by exploiting representations extracted from very long-term historical time series.
Takeaways & Limitations
Learning long-term temporal patterns requires longer sequences, which increase model complexity and motivate efficiency measures such as fewer Transformer blocks.

Abstract

from arXiv · show

Multivariate Time Series (MTS) forecasting plays a vital role in a wide range of applications. Recently, Spatial-Temporal Graph Neural Networks (STGNNs) have become increasingly popular MTS forecasting methods. STGNNs jointly model the spatial and temporal patterns of MTS through graph neural networks and sequential models, significantly improving the prediction accuracy. But limited by model complexity, most STGNNs only consider short-term historical MTS data, such as data over the past one hour. However, the patterns of time series and the dependencies between them (i.e., the temporal and spatial patterns) need to be analyzed based on long-term historical MTS data. To address this issue, we propose a novel framework, in which STGNN is Enhanced by a scalable time series Pre-training model (STEP). Specifically, we design a pre-training model to efficiently learn temporal patterns from very long-term history time series (e.g., the past two weeks) and generate segment-level representations. These representations provide contextual information for short-term time series input to STGNNs and facilitate modeling dependencies between time series. Experiments on three public real-world datasets demonstrate that our framework is capable of significantly enhancing downstream STGNNs, and our pre-training model aptly captures temporal patterns.

1 INTRODUCTION

Multivariate forecasting requires modeling both long-term temporal patterns and dependencies among variables, but STGNNs often rely on short windows. STEP addresses these limitations by pre-training on long histories and supplying contextual representations to downstream STGNNs.

Motivation: Multivariate time series exhibit multiple periodicities and interdependencies, so accurate forecasting requires both temporal and spatial modeling.Traffic sensors can show daily and weekly patterns while sharing trends because of network connectivity.
Existing Methods: STGNNs combine graph neural networks for inter-series dependencies with sequential models for temporal patterns.Their joint spatial-temporal modeling has achieved state-of-the-art performance.
Limitations: Short input windows leave STGNNs without broader context and can make both future trends and dependency graphs difficult to infer.Similar short windows may correspond to different contexts, while short-term observations may not reflect inter-series similarity.
STEP Framework: STEP enhances STGNNs with a scalable pre-training model that learns temporal patterns from very long histories and generates contextual segment-level representations.These representations are designed to address the missing-context challenge for short-term STGNN inputs.
STEP Framework: TSFormer uses Transformer blocks and masked autoencoding, while a graph structure learner derives dependency graphs and uses a kNN graph as regularization.The framework is intended to extend to almost arbitrary STGNNs.
Evaluation: Experiments on three real-world datasets show significant enhancement of downstream STGNN performance, while TSFormer captures temporal patterns.The contribution statement reports both downstream gains and successful temporal-pattern capture.

2 PRELIMINARIES

The paper formalizes multivariate time series as variables with temporal observations and interdependencies represented by a graph. Forecasting uses historical signals to predict values at the nearest future time steps.

Multivariate Time Series: A multivariate time series contains multiple time-dependent variables, represented as X ∈ R^T×N×C.T denotes time steps, N variables, and C channels.
Dependency Graph: A dependency graph represents how each variable depends on its own history and other variables.The graph G = (V, E) has one node per variable and can be represented by an adjacency matrix A ∈ R^N×N.
Forecasting Problem: Multivariate time series forecasting maps historical signals X ∈ R^T_h×N×C to predictions Y ∈ R^T_f×N×C for the nearest future steps.The history spans T_h steps and the forecast spans T_f steps.

3 MODEL ARCHITECTURE

STEP uses an efficient masked-autoencoding Transformer to learn segment-level representations from very long time-series histories, then uses those representations to enhance downstream STGNN forecasting. Its architecture addresses the need for long-range temporal context while supporting graph-based dependency learning.

Framework Overview: STEP has separate pre-training and forecasting stages: TSFormer learns from segmented long-term histories, while an enhanced STGNN uses its representations.The pre-training model is trained with masked autoencoding, and the forecasting stage incorporates the resulting segment-level representations.
Pre-training Stage: Time-series semantics emerge at the segment level, so TSFormer divides signals into patches rather than modeling isolated points.Patches also match the segment inputs used by downstream STGNNs.
Pre-training Stage: TSFormer is designed for long sequences because temporal patterns such as weekly periodicity may require thousands of time slices to observe.For five-minute traffic data, the paper gives 2016 consecutive slices as the minimum example for learning weekly periodicity.
Pre-training Stage: TSFormer reconstructs partially observed signals with an asymmetric masked autoencoder whose encoder processes only visible patches and whose lightweight decoder reconstructs the full signal.A high masking ratio and fewer Transformer blocks reduce pre-training computation.
Forecasting Stage: During forecasting, the pre-trained encoder provides context for the latest STGNN input, while representations also support learning a discrete sparse dependency graph.The architecture can extend to different STGNN backends; Graph WaveNet is used as a representative backend.

4 EXPERIMENTS

Experiments evaluate STEP on three real-world traffic datasets, using long historical windows for TSFormer pre-training and short-horizon forecasting. STEP consistently improves Graph WaveNet and TSFormer representations reveal meaningful periodic structure.

4.1 Experimental Setup: Experiments use METR-LA, PEMS-BAY, and PEMS04, covering traffic speed or flow from hundreds of sensors recorded every five minutes.The datasets span different sensor counts, collection periods, and geographic settings.
4.1 Experimental Setup: TSFormer uses 168 patches for METR-LA and PEMS-BAY and 336 for PEMS04, covering one or two weeks of historical data.Each patch has length 12; the forecasting target is the next 12 time steps.
4.2 Main Results: STEP consistently achieves the best performance across almost all horizons and datasets in the forecasting experiments.The evaluation compares STEP with STGNN baselines, including GTS, MTGNN, and GMAN.
4.2 Main Results: STEP provides stable performance gains for Graph WaveNet by exploiting TSFormer representations learned from very long-term historical time series.The framework is designed to supply contextual information beyond the short input window.
4.3 Inspecting The TSFormer: On PEMS04, TSFormer identifies similar patches, with pairwise representation similarities revealing clear daily and weekly periodicities.The analysis uses 336 patches covering the previous two weeks and visualizes their pairwise cosine similarities.

PEMS-BAY

The ablation study evaluates STEP’s graph structure learning, long-sequence representations, and transfer to another STGNN backend. Hyper-parameter experiments examine masking ratio and kNN graph size, identifying trade-offs around information loss and redundancy.

Ablation Study: STEP outperforms STEP w/o reg, supporting the superiority of TSFormer’s long-sequence representations for improving graph quality.
Ablation Study: STEP significantly enhances DCRNN, demonstrating generality across a seq2seq-based downstream STGNN.
Ablation Study: STEP outperforms STEP w/o GSL, indicating that graph structure learning contributes positively to performance.
Ablation Study: STEP w/o GSL still achieves satisfactory performance, demonstrating the importance of segment-level representations.
Hyper-parameter Study: Both masking ratio r and kNN graph size k have optimal values in the METR-LA experiments.
Hyper-parameter Study: Small masking ratios create trivial reconstruction tasks, whereas large ratios remove excessive information needed to learn temporal patterns.

5 RELATED WORK

Related work positions STGNNs as joint spatial-temporal models and reviews graph-structure learning approaches. It also connects STEP’s pre-training design to Transformer-based self-supervised representation learning.

STGNNs: STGNNs combine graph neural networks and sequential models to jointly model spatial and temporal dependencies in multivariate time series.
Graph Structure Learning: Recent methods learn graph structures jointly with graph neural networks to model dependencies between nodes.
Pre-training: Pre-training models learn representations from massive unlabeled data for use in downstream tasks, with BERT and GPT as prominent Transformer-based examples.
Pre-training: MAE uses masked autoencoding to train models self-supervised, efficiently, and without the massive labeled data required by supervised pre-training.

6 CONCLUSION

The conclusion presents STEP as a framework addressing STGNNs’ limited access to long-term information. It combines TSFormer representations with downstream STGNNs and reports superiority across three real-world datasets.

6 CONCLUSION: STEP addresses the inability of STGNNs to learn long-term information in multivariate time series forecasting.
6 CONCLUSION: TSFormer efficiently learns temporal patterns from very long-term historical time series and generates segment-level representations.
6 CONCLUSION: Experiments on three real-world datasets show the superiority of STEP and TSFormer.

A.1 Baseline Details

The baseline details include classical statistical, machine-learning, and recurrent neural forecasting models used for comparison.

A.1 Baseline Details: HA models time series as periodic processes and predicts future values using weighted averages from previous periods.
A.1 Baseline Details: VAR assumes past time series are stationary and estimates relationships between series and their lagged values.
A.1 Baseline Details: SVR applies a linear support vector machine to the classical time series regression task.

A.2 Optimization Settings

The optimization settings specify separate pre-training and forecasting configurations, including initialization, scaling, curriculum learning, and masking-ratio efficiency analysis.

Pre-training setting: Table 3 presents the pre-training setting, including positional-embedding and mask-token initialization choices.Positional embeddings use a uniform distribution; the mask token uses a truncated normal distribution with μ=0 and σ=0.02.
Pre-training setting: The pre-training stage scales learning rate with batch size using lr = base_lr × (batch_size/8).
Forecasting setting: Table 4 presents forecasting settings that use curriculum learning, warm-up, and epoch-dependent decay of λ.Prediction length increases by one every cl_num epochs after warm_num warm-up epochs, while λ=1/(⌈epoch/6⌉).
Efficiency setting: Figure 5 compares training speed across masking ratios r, with higher masking ratios making TSFormer more efficient.

B EFFICIENCY

The efficiency analysis compares training speed across masking ratios, methods, and STEP forecasting variants, emphasizing TSFormer’s lightweight pre-training and preprocessing-based savings.

Experimental setup: The efficiency experiments measure average training time per epoch on METR-LA using a fixed server and RTX 3090 GPUs.
Method comparison: Figure 4 compares training speed across different methods.
Pre-training efficiency: Higher masking ratios improve TSFormer efficiency, while its lightweight architecture enables training on a single NVIDIA 3090 GPU.The stated design factors are a high masking ratio and fewer Transformer blocks in the encoder and decoder.
Forecasting efficiency: During forecasting, fixed TSFormer parameters allow preprocessing off-the-shelf representations to reduce redundant training calculations.STEP without preprocessing is evaluated as the STEP w/o pre variant.

C VISUALIZATION

Visualization experiments inspect TSFormer reconstructions and forecasting behavior, showing accurate reconstruction from limited context and predictions across selected traffic series.

Reconstruction: TSFormer accurately reconstructs masked patches on PEMS04 even with very limited information surrounding the unmasked patches.The reconstruction results are presented in Figure 7 and are used to assess learned temporal patterns from very long-term series.
Forecasting: Figure 8 visualizes forecasts and ground truth for six randomly selected METR-LA time series from June 13th to June 16th, 2012.
Forecasting: The model makes accurate predictions for different time series and does not overfit an apparent traffic-sensor failure in the afternoon of June 13th, 2012.