PaperlayerView exact revision on arXiv

Source-linked AI summary

Scalable Spatiotemporal Graph Neural Networks

Andrea Cini, Ivan Marisca, Filippo Maria Bianchi, Cesare Alippi

arXiv:2209.06520v2cs.LGcs.AI

TL;DR

Spatiotemporal GNNs pose scalability challenges for large graphs and long sequences, while scalable methods have received less attention for dynamic data. The paper proposes a randomized spatiotemporal encoder with node-wise decoder training. It reports state-of-the-art-competitive forecasting and improved scalability, while noting that processed sequences can require substantial system memory.

  • Problem

    Scalable methods have received limited attention for spatiotemporal GNNs, whose computational costs can hinder use on large graphs and long sequences.

    Source: §1 · p. 1

  • Method

    A randomized recurrent encoder and graph-adjacency powers produce spatiotemporal node embeddings that a trainable decoder processes with node-wise mini-batches.

    Source: §1 · p. 2

  • Results

    SGP competes with the state of the art on popular medium-sized benchmarks while greatly improving scalability in large sensor networks.

    Source: §1 · p. 2, §6 · p. 7

  • Takeaways & Limitations

    SGP offers a scalable approach for graph-based spatiotemporal forecasting, with preprocessing that can be distributed and stored on disk for batched training.

    Source: §6 · p. 7

    The processed sequence can occupy a large portion of system memory, depending on reservoir size.

    Source: §6 · p. 7

Abstract

from arXiv · show

Neural forecasting of spatiotemporal time series drives both research and industrial innovation in several relevant application domains. Graph neural networks (GNNs) are often the core component of the forecasting architecture. However, in most spatiotemporal GNNs, the computational complexity scales up to a quadratic factor with the length of the sequence times the number of links in the graph, hence hindering the application of these models to large graphs and long temporal sequences. While methods to improve scalability have been proposed in the context of static graphs, few research efforts have been devoted to the spatiotemporal case. To fill this gap, we propose a scalable architecture that exploits an efficient encoding of both temporal and spatial dynamics. In particular, we use a randomized recurrent neural network to embed the history of the input time series into high-dimensional state representations encompassing multi-scale temporal dynamics. Such representations are then propagated along the spatial dimension using different powers of the graph adjacency matrix to generate node embeddings characterized by a rich pool of spatiotemporal features. The resulting node embeddings can be efficiently pre-computed in an unsupervised manner, before being fed to a feed-forward decoder that learns to map the multi-scale spatiotemporal representations to predictions. The training procedure can then be parallelized node-wise by sampling the node embeddings without breaking any dependency, thus enabling scalability to large networks. Empirical results on relevant datasets show that our approach achieves results competitive with the state of the art, while dramatically reducing the computational burden.

1 Introduction

The paper addresses the limited attention paid to scalability for spatiotemporal GNNs, where preserving temporal and spatial dependencies complicates scalable training. It proposes a randomized spatiotemporal encoder with node-wise training and reports state-of-the-art-competitive performance alongside computational efficiency and scalability.

1 Introduction

  • Subsampling can disrupt spatial and temporal dependencies, while precomputing aggregated graph features is not straightforward to extend to spatiotemporal data.The challenge is to account for temporal dependencies as well as graph topology during preprocessing.Source: §1 · p. 1
  • A training-free randomized recurrent encoder embeds temporal history, then graph-adjacency powers build node representations of spatiotemporal dynamics for a trainable decoder.The decoder is designed as filters localized at different spatiotemporal scales.Source: §1 · p. 2
  • Training mini-batches can be sampled independently of sequence length and graph size, making training complexity O(1) in the spatiotemporal dimension.The passage contrasts this with O(TE) for standard spatiotemporal GNNs and O(T^2E) for attention-based architectures.Source: §1 · p. 2
  • The proposed framework targets multivariate time-series forecasting where channels have spatial relationships described by a graph.Source: §1 · p. 2
  • The authors introduce two benchmarks for scalable spatiotemporal forecasting architectures and report an extensive empirical evaluation of their architecture and variations.They frame the need in the context of large graphs such as large sensor networks.Source: §1 · pp. 1–2
  • SGP performs on par with the state of the art while being computationally efficient and extremely scalable.The authors also describe it as easy to implement.Source: §1 · p. 2

2 Preliminaries and Problem Definition

The section formalizes forecasting on discrete-time spatiotemporal graphs and describes echo-state networks as randomized recurrent encoders of input history. Their reservoir states provide high-dimensional embeddings that can support downstream forecasting.

2 Preliminaries and Problem Definition

  • Spatiotemporal forecasting predicts the next H observations from a window of W past measurements.Source: §2 · p. 2
  • An echo-state network uses random recurrent connections to encode input history in a high-dimensional state representation for a trainable readout.The input is fed into a randomized nonlinear reservoir, whose internal state embeds the input dynamics.Source: §2 · p. 2
  • With properly defined random matrices, reservoir states become informative embeddings of the input time-series history.A linear readout can be optimized by least squares for tasks including time-series forecasting.Source: §2 · p. 2

3 Scalable Spatiotemporal GNNs

SGP combines a training-free temporal-and-spatial encoder with a structured decoder for forecasting. Its multi-scale representations support node-wise decoder training and H-step-ahead predictions.

3.1 Spatiotemporal Encoder

  • A training-free DeepESN builds hierarchical temporal encodings, then graph-shift powers propagate them across spatial scales before concatenation.For directed graphs, the method can use both adjacency orientations; graph-order features can be computed recursively or precomputed offline.Source: §3 · p. 3, §3.1 · p. 3

3.2 Multi-Scale Decoder

  • The decoder’s sparse first layer exploits the embedding’s spatial and temporal structure, with grouped 1-D convolutions parallelizing computation and reducing parameters.Separate weights localize filters by spatial order and temporal scale; the resulting representations feed an MLP for H-step-ahead predictions.Source: §3.2 · p. 4
  • Because embeddings already contain spatiotemporal relationships, decoder training can independently sample node-time points without collecting further context.The encoder needs neither training nor supervision, and embeddings can be computed before decoder training.Source: §3 · p. 3, §3.2 · p. 4, §3.1 · p. 4

4 Related works

The related work distinguishes scalable static-graph methods from approaches for dynamic spatiotemporal graphs. SGP is positioned among time-then-graph models, with contrasts to SIGN and DynGESN.

4 Related works

  • SGP is a time-then-graph model; prior work includes recurrent message passing, alternating temporal and spatial convolutions, and attention-based propagation.Full spatial Transformer-like attention makes computation scale quadratically with node count; cited work reports greater expressiveness for time-then-graph models than alternating processing.Source: §4 · p. 4
  • SIGN preprocesses propagation across graph neighborhoods before an MLP, but is limited to static graphs and raw node attributes.The passage identifies SIGN as the closest static-graph approach to SGP.Source: §4 · p. 4
  • Table 1 reports traffic-benchmark MAE, MSE, and MAPE over a one-hour horizon, plus MAE at 15-, 30-, and 60-minute horizons.Results are averaged over three independent runs.Source: Table 1 · p. 5
  • DynGESN also processes dynamical graphs with randomized recurrence, whereas SGP combines a randomized encoder with a trainable decoder.Source: §4 · p. 5

5 Empirical evaluation

The evaluation tests SGP on medium-scale traffic benchmarks and larger energy datasets, assessing forecasting accuracy and scalability. SGP is among the strongest traffic forecasters and remains competitive on large datasets while training with higher update frequency and contained GPU memory use.

5 Empirical evaluation

  • The evaluation includes METR-LA and PEMS-BAY traffic data, plus CER-E with 6,435 meters and PV-US with 5,016 simulated photovoltaic farms.The large energy datasets are at least an order of magnitude larger than datasets typically used for spatiotemporal forecasting benchmarks.Source: §5 · p. 5

5.1 Results

  • The large-scale results are reported as MAE across three forecast horizons, alongside timing and memory consumption, with asterisks marking cases requiring edge subsampling.The caption says results are averaged over at least three independent runs.Source: Table 2 · p. 6

5 Empirical evaluation

  • Large-scale experiments assess scalability rather than maximizing accuracy, using six months of data, three forecast horizons, and graphs with either at most 100 neighbors per node or unconstrained density.Models train for one hour under a 12 GB GPU-memory budget; SGP preprocesses embeddings before decoder training.Source: Baselines · p. 6

5.1 Results

  • SGP is among the best-performing architectures on all considered traffic scenarios, while its preprocessing amortizes the graph-convolution complexity of some baselines.The full-attention baseline is the strongest competitor but scales quadratically with node count.Source: §5.1 · p. 7
  • On large-scale datasets, SGP’s update frequency reaches up to two orders of magnitude higher, with constant resource utilization and competitive forecasting accuracy.The comparison uses a 12 GB GPU-memory budget; nearly all baselines require edge subsampling to meet it.Source: Table 2 · p. 6, §5.1 · p. 7

6 Remarks and conclusion

SGP competes with state-of-the-art methods on medium-sized benchmarks and improves scalability on large sensor networks. Preprocessed sequences can strain system memory, though disk storage and batch loading are possible; future work could integrate spatial and temporal encoding more tightly and test larger benchmarks.

6 Remarks and conclusion

  • SGP competes with the state of the art on medium-sized benchmarks while improving scalability on large sensor networks.Sampling reduces GPU memory use, but processed sequences may occupy substantial system memory depending on reservoir size; preprocessing can be distributed and data stored on disk for batch loading.Source: §6 · p. 7
  • Future work could more tightly integrate spatial and temporal encoding and assess performance on larger benchmarks.Source: §6 · p. 7

A Detailed experimental settings

The appendix provides additional details on the experiments’ settings.

A Detailed experimental settings

A.1 Software platform

The experimental code is publicly available and uses several open-source libraries and Neptune for experiment logging.

A.1 Software platform

  • The code used for the experiments is available in the project’s GitHub repository.Source: §A.1 · p. 9
  • The experiments use PyTorch, PyTorch Geometric, Torch Spatiotemporal, PyTorch Lightning, and NumPy.Source: §A.1 · p. 9
  • Neptune supports experiment logging, and baselines use their open-source implementations where available.Source: §A.1 · p. 9

A.2 Hardware platform

Scalability experiments were run on a two-processor AMD EPYC 7513 server with four NVIDIA RTX A5000 GPUs. Update-step timing used a controlled number of model updates for reproducibility.

A.2 Hardware platform

  • Scalability experiments ran on a server with two AMD EPYC 7513 processors and four NVIDIA RTX A5000 GPUs.Timings were taken for each model's update step, with the number of updates set accordingly to support reproducibility.Source: §A.2 · p. 9
  • Update-step timings used a matched number of updates per model to support reproducibility.The passage specifies that timings were taken for each model's update step and the update count was set accordingly.Source: §A.2 · p. 9
  • The passage identifies the hardware platform but does not report its operating system or other software specifications.This bounds what can be inferred about the experimental environment from this section.Source: §A.2 · p. 9

A.3 Datasets

The appendix describes dataset construction, SGP architecture details, and training and evaluation procedures across forecasting and scalability experiments.

A.3 Datasets

  • Input graphs are formed from spatial information or time-series similarity, then sparsified by thresholding and optionally limiting each node's neighbors; time of day is the only exogenous variable.PV-US uses geographic coordinates with a Gaussian kernel, while CER-En uses correntropy among time series.Source: §A.3 · pp. 9–10
  • The experiments cover traffic, household and enterprise energy consumption, and simulated photovoltaic production datasets, with differing temporal aggregation intervals and network sizes.Traffic observations are aggregated every 5 minutes; CER-En records 6,435 meters at 30-minute intervals, while PV-US includes 5,016 farms aggregated to 30-minute means.Source: §A.3 · p. 10

A.4 Additional details on SGP architecture

  • The DeepESN reservoir uses progressively reduced discount factors, 30% zeroed weights, tanh activations, and recurrent weights normalized to spectral radius below one.The initial discount factor is reduced by 0.1 at each layer.Source: §A.4 · p. 10
  • Spatial embeddings are computed iteratively and concatenated with the graph-wise average temporal embedding; the MLP decoder uses residual connections, SiLU, and optional dropout.The graph-wise average serves as a global attribute.Source: §A.4 · p. 10

A.5 Training and evaluation procedure

  • Traffic SGP settings use dataset-specific reservoirs and K = 4 bidirectional encoding, while baselines generally follow prior-work settings and are evaluated using validation-selected weights.The passage also describes baseline architecture choices, validation-based early stopping, and optimizer settings for traffic experiments.Source: Traffic · p. 10, Large-scale · p. 11
  • Scalability timing averages the middle 140 of 150 batch measurements and excludes per-batch edge-subsampling overhead; GPU memory is monitored with NVIDIA System Management Interface.For large-scale experiments, SGP uses K = 2 and unidirectional encoding to limit storage to approximately 80 GB per dataset.Source: Large-scale · pp. 10–11

Related papers

Loading 2209.06520v2…