PaperlayerView exact revision on arXiv

Source-linked AI summary

Spatial-Temporal Transformer Networks for Traffic Flow Forecasting

Mingxing Xu, Wenrui Dai, Chunmiao Liu, Xing Gao, Weiyao Lin, Guo-Jun Qi, Hongkai Xiong

arXiv:2001.02908v2eess.SPcs.LG

TL;DR

Long-term traffic forecasting is difficult because traffic exhibits nonlinear, dynamic spatial-temporal dependencies and autoregressive prediction can accumulate errors. STTNs combine dynamic directed spatial modeling with long-range temporal modeling, achieving state-of-the-art performance on two real-world datasets, especially for long-term forecasts.

  • Problem

    Traffic forecasting lacks accurate, timely long-term prediction methods that adequately model nonlinear, dynamic spatial-temporal dependencies and avoid accumulated autoregressive errors.

  • Method

    STTNs combine a spatial transformer for time-varying directed spatial dependencies with a temporal transformer for long-range temporal dependencies in spatial-temporal blocks.

  • Results

    STTNs achieve state-of-the-art traffic forecasting performance on PeMSD7(M) and PeMS-BAY, especially for long-term predictions.

  • Takeaways & Limitations

    The spatial transformer dynamically models various spatial dependency scales, and its design can generalize to dynamical graph feature learning applications.

Abstract

from arXiv · show

Traffic forecasting has emerged as a core component of intelligent transportation systems. However, timely accurate traffic forecasting, especially long-term forecasting, still remains an open challenge due to the highly nonlinear and dynamic spatial-temporal dependencies of traffic flows. In this paper, we propose a novel paradigm of Spatial-Temporal Transformer Networks (STTNs) that leverages dynamical directed spatial dependencies and long-range temporal dependencies to improve the accuracy of long-term traffic forecasting. Specifically, we present a new variant of graph neural networks, named spatial transformer, by dynamically modeling directed spatial dependencies with self-attention mechanism to capture realtime traffic conditions as well as the directionality of traffic flows. Furthermore, different spatial dependency patterns can be jointly modeled with multi-heads attention mechanism to consider diverse relationships related to different factors (e.g. similarity, connectivity and covariance). On the other hand, the temporal transformer is utilized to model long-range bidirectional temporal dependencies across multiple time steps. Finally, they are composed as a block to jointly model the spatial-temporal dependencies for accurate traffic prediction. Compared to existing works, the proposed model enables fast and scalable training over a long range spatial-temporal dependencies. Experiment results demonstrate that the proposed model achieves competitive results compared with the state-of-the-arts, especially forecasting long-term traffic flows on real-world PeMS-Bay and PeMSD7(M) datasets.

1 INTRODUCTION

Traffic forecasting must handle nonlinear, time-varying spatial dependencies and long-range temporal dependencies, particularly for accurate long-term prediction. STTNs address these challenges with dynamic spatial and temporal transformers and achieve strong long-term results on two real-world datasets.

  • Accurate, timely traffic forecasting is a core intelligent transportation system requirement, but long-term prediction remains challenging.
  • Spatial dependencies vary across time because of road topology, traffic speeds, weather, rush hours, and accidents.
  • Autoregressive methods can propagate and accumulate prediction errors during long-term forecasting.
  • The proposed spatial-temporal blocks dynamically model dependencies, and experiments on PeMSD7(M) and PeMS-BAY show especially strong performance for long-term predictions.
  • STTNs use a spatial transformer for time-varying directed dependencies and a temporal transformer for multi-step prediction from long-range dependencies.

2 RELATED WORK

Prior traffic forecasting methods include statistical, neural, convolutional, graph, and sequential models, but they face limitations in nonlinear modeling, topology preservation, long-range dependence, training time, or scalability.

  • Statistical models capture uncertainty but their linear nature limits modeling of highly nonlinear traffic flows.
  • CNN-based methods extract spatial features on regular grids, but grid conversion loses inherent topology information from irregular traffic networks.
  • RNNs, GRUs, and LSTMs address long-range dependencies but can suffer from gradient, sequence-length, training-time, and scalability limitations.
  • Convolutional sequence models require multiple layers for large contexts, while WaveNet’s hidden-layer count increases linearly with input-sequence length.

3 PROPOSED MODEL

The proposed model is a spatial-temporal transformer network that combines spatial and temporal transformers within stacked blocks, followed by a prediction layer.

  • The model contains two main components: stacked spatial-temporal blocks and a prediction layer.
  • Each spatial-temporal block uses spatial and temporal transformers to jointly extract spatial-temporal features.

3.1 Problem Formulation

Traffic forecasting is formulated on a sensor graph using historical traffic conditions to predict multiple future conditions. The model learns dynamic spatial and long-range temporal dependencies while making all future predictions from error-free history.

  • A traffic network is represented as a graph whose nodes are sensors and whose edges encode physical connectivity and distance-based adjacency.
  • Given M historical traffic conditions and graph G, the forecasting model predicts T future traffic conditions for the sensors.
  • The forecasting function captures dynamical spatial dependencies from historical observations and the traffic network.
  • A temporal transformer efficiently learns long-term temporal dependencies using self-attention.
  • The formulation simultaneously predicts all T future conditions from error-free historical observations to address error propagation.

3.2 Overall Architecture

STTN stacks spatial-temporal blocks that jointly extract dynamical spatial and temporal features before a convolutional prediction layer produces multi-step forecasts.

  • Overall architecture: Each spatial-temporal block combines one spatial transformer and one temporal transformer to jointly model traffic dependencies.
  • Overall architecture: The spatial transformer extracts spatial features from node features and the graph adjacency matrix, then passes them to the temporal transformer.
  • Overall architecture: Residual connections are used for stable training, and multiple spatial-temporal blocks can be stacked to increase model capacity.
  • Prediction layer: The prediction layer applies two convolutional layers to the final block's spatial-temporal features to forecast T future traffic conditions for N nodes.
  • Prediction layer: Mean absolute loss is used to train the model.

3.3 Spatial Transformer

The spatial transformer combines fixed topology-based and dynamically learned graph convolutions, using attention to model time-varying directed dependencies and multiple spatial patterns.

  • Architecture: The spatial transformer includes positional embedding, fixed graph convolution, dynamical graph convolution, and gated information fusion.
  • Spatial-temporal positional embedding: Learnable spatial and temporal positional embeddings inject topology, connectivity, and time-step information into node inputs.
  • Fixed graph convolutional layer: Fixed graph convolution uses the road topology to capture stationary spatial dependencies determined by sensor connectivity and distance.
  • Dynamical graph convolutional layer: The dynamical graph convolution layer projects node features into latent subspaces and uses self-attention to learn time-varying directed spatial dependencies.
  • Dynamical graph convolutional layer: Multi-head attention learns multiple spatial dependency patterns from different latent subspaces, while a feed-forward network further updates node features.
  • Feature fusion: A gate fuses features from fixed and dynamical graph convolutional layers, and the spatial transformer's output is passed to the temporal transformer.
  • General dynamical graph neural networks: The spatial transformer can be viewed as a message-passing dynamical graph neural network that updates each node after aggregating messages from other nodes.

3.4 Temporal Transformers

The temporal transformer uses self-attention to capture long-range bidirectional dependencies across time, enabling parallel multi-step prediction from historical observations.

  • 3.4 Temporal Transformers: The temporal transformer processes nodes in parallel and can scale to longer sequences by increasing the sliding-window length M.Parallel processing avoids the recurrent structure's gradient issues and preserves computation efficiency for long sequences.
  • 3.4 Temporal Transformers: Self-attention dynamically computes temporal dependencies in high-dimensional latent query, key, and value subspaces.Learned linear mappings produce the query, key, and value representations used by scaled dot-product attention.
  • 3.4 Temporal Transformers: Multi-step predictions for vτ+1 through vτ+T are made simultaneously from historical observations vτ−M+1 through vτ.The approach avoids basing later predictions on previously predicted values.
  • 3.4 Temporal Transformers: A feed-forward network, residual connection, and output projection transform attended temporal features into the temporal transformer's output.The residual connection is adopted for stable training.
  • 3.4 Temporal Transformers: The temporal transformer models long-range dependencies by allowing each time step to attend to the remaining steps within a sliding window.This bidirectional attention contrasts with the preceding-step limitation of RNN-based models.

4 EXPERIMENTS

Experiments evaluate STTN on two real-world traffic datasets against conventional and graph-based baselines. STTN is especially competitive for long-term forecasting and captures changing traffic patterns over extended horizons.

  • 4 EXPERIMENTS: Ablation studies examine multi-step prediction and the effectiveness of the spatial and temporal transformers for long-term traffic forecasting.The study also analyzes blocks, feature channels, layers, and attention heads.
  • 4 EXPERIMENTS: STTN is evaluated on PeMSD7(M) and PEMS-BAY, using traffic speeds aggregated every five minutes and graph topology information.The datasets contain 228 and 325 sensors, respectively, and PEMS-BAY uses a directed graph.
  • 4 EXPERIMENTS: The evaluation compares STTN with historical, statistical, linear, neural, recurrent, graph-convolutional, and Graph WaveNet baselines using MAE, MAPE, and RMSE.Table 1 reports forecasts at multiple horizons for both datasets.
  • 4.4 Experimental Results: On PeMSD7(M), STTN outperforms STGCN and DCRNN by a margin that grows with the prediction horizon, while outperforming Graph WaveNet for long-term prediction.It remains competitive with Graph WaveNet for short-term prediction.
  • 4.4 Experimental Results: On PEMS-BAY, STTN is competitive with Graph WaveNet and outperforms STGCN and DCRNN.STTN learns dynamical directed spatial dependencies from a symmetric adjacency matrix without prior upstream/downstream information.
  • 4.4 Experimental Results: In one-day PeMSD7(M) visualizations, STTN and Graph WaveNet improve forecasting in changing areas, while STTN captures continuous changes across τ ∈[84, 192].STGCN and DCRNN show evident time shifts, especially where traffic changes sharply.

4.5 Computational Complexity

STTN is designed to remain computationally efficient as forecasting horizons grow. Compared with Graph WaveNet and DCRNN, it reduces reported computational costs while supporting long-term prediction.

  • 4.5 Computational Complexity: STTN reduces computational costs by 10-40% versus Graph WaveNet and 40-60% versus DCRNN.The comparison uses average training time per epoch on the same GPU.
  • 4.5 Computational Complexity: DCRNN training time increases with the number of prediction time steps because its recurrent structure uses joint loss for multiple steps.STGCN is efficient because it uses fully convolutional structures.
  • 4.5 Computational Complexity: Table 2 reports average training time in seconds per epoch for STTN, Graph WaveNet, STGCN, and DCRNN on PEMS-BAY and PeMSD7(M).The table organizes computational comparison by dataset and model.
  • 4.5 Computational Complexity: Table 3 compares STGCN and STTN under autoregressive and multi-step prediction using MAE, MAPE, and RMSE on PeMSD7(M).This comparison evaluates the prediction strategy alongside computational considerations.

4.6 Ablation Studies

The ablations show that dynamic spatial dependencies and long-range temporal dependencies improve long-term traffic forecasting. Spatial transformers benefit from global modeling and more attention heads, while temporal transformers outperform fixed graph convolution for long-term prediction.

  • Spatial Transformer: STTN-S(1,1) outperforms the fixed-dependency baseline, especially for long-term traffic prediction.The improvement is particularly evident in sharply changing areas during one-day forecasting.
  • Spatial Transformer: Global dynamical spatial dependencies outperform local dependencies for traffic flow forecasting.The local variant restricts attention to k-nearest neighbors, whereas the proposed spatial transformer models global dependencies.
  • Spatial Transformer: Increasing spatial-transformer attention heads continuously improves forecasting by modeling dependencies in different latent subspaces.Increasing hidden layers also benefits spatial-transformer performance, although the passage contrasts this with the stronger role of attention heads.
  • Temporal Transformer: Larger temporal convolution kernels improve long-term prediction, and the temporal transformer outperforms fixed graph convolution.The comparison uses convolution kernel sizes 3, 6, 9, and 12, with the temporal transformer evaluated against the fixed-graph baseline.
  • Temporal Transformer: Multi-head attention does not benefit temporal forecasting as much as spatial forecasting, while increasing temporal hidden layers tends to improve performance.The ablation attributes the difference to temporal dependencies being less complex than spatial dependencies.

4.7 Model Configurations

Model configuration experiments show that increasing capacity generally improves STTN until additional spatial-temporal blocks become stable. Jointly enhancing the spatial and temporal transformers is preferred for accurate prediction.

  • Spatial-Temporal Blocks: MAE decreases when multiple spatial-temporal blocks are cascaded, then stabilizes once enough blocks are stacked.The configurations are evaluated on PeMSD7(M).
  • Feature Capacity: Higher feature dimensions exploit more information for accurate prediction.Feature-channel count determines the dimension of the latent subspace used to dynamically compute dependencies.
  • Transformer Capacity: Increasing hidden layers benefits the spatial transformer more than the temporal transformer, indicating that spatial capacity affects long-term prediction.The configuration study suggests jointly enhancing spatial and temporal transformer capacity.

5 CONCLUSION

The paper proposes spatial-temporal transformer networks that dynamically model multi-scale spatial dependencies and long-range temporal dependencies for long-term traffic-flow prediction. Experiments on two real-world datasets report superior performance, especially for long-term forecasting.

  • Conclusion: STTNs dynamically model various spatial-dependency scales and capture long-range temporal dependencies.The spatial transformer is also presented as potentially generalizable to dynamical graph feature learning.
  • Conclusion: Experiments on two real-world datasets demonstrate superior STTN performance, especially for long-term traffic prediction.The paper identifies further investigation of dynamical graph feature learning as future work.
Loading 2001.02908v2…