PaperlayerView exact revision on arXiv

Source-linked AI summary

Learning dynamic and hierarchical traffic spatiotemporal features with Transformer

Haoyang Yan, Xiaolei Ma

arXiv:2104.05163v1cs.AI

TL;DR

Long-term, network-wide traffic speed forecasting is difficult because fixed adjacent matrices may not capture actual, changing road-network dependencies. Traffic Transformer adapts Transformer attention to learn and fuse dynamic hierarchical spatial-temporal features, achieving state-of-the-art results on three real-world datasets while revealing influential nodes. Its scope is constrained by quadratic complexity and limited sensor counts.

  • Problem

    Long-term, network-wide traffic speed forecasting is challenging, while GCN-based models rely on predefined fixed adjacent matrices that inadequately reflect dynamic traffic dependencies.

  • Method

    Traffic Transformer uses multi-head and masked multi-head attention to dynamically extract hierarchical global and local spatial-temporal features and fuse them for forecasting.

  • Results

    Traffic Transformer achieves state-of-the-art results on three real-world datasets and helps identify hierarchical influential nodes in the network.

  • Takeaways & Limitations

    Learned spatial dependencies can improve forecasting when predefined adjacency is unreliable, while predefined adjacency may still help in moderately complex networks.

  • Takeaways & Limitations

    The study uses fewer than 300 sensors because Transformer’s O(n^2) complexity makes training difficult for larger sensor networks.

Abstract

from arXiv · show

Traffic forecasting is an indispensable part of Intelligent transportation systems (ITS), and long-term network-wide accurate traffic speed forecasting is one of the most challenging tasks. Recently, deep learning methods have become popular in this domain. As traffic data are physically associated with road networks, most proposed models treat it as a spatiotemporal graph modeling problem and use Graph Convolution Network (GCN) based methods. These GCN-based models highly depend on a predefined and fixed adjacent matrix to reflect the spatial dependency. However, the predefined fixed adjacent matrix is limited in reflecting the actual dependence of traffic flow. This paper proposes a novel model, Traffic Transformer, for spatial-temporal graph modeling and long-term traffic forecasting to overcome these limitations. Transformer is the most popular framework in Natural Language Processing (NLP). And by adapting it to the spatiotemporal problem, Traffic Transformer hierarchically extracts spatiotemporal features through data dynamically by multi-head attention and masked multi-head attention mechanism, and fuse these features for traffic forecasting. Furthermore, analyzing the attention weight matrixes can find the influential part of road networks, allowing us to learn the traffic networks better. Experimental results on the public traffic network datasets and real-world traffic network datasets generated by ourselves demonstrate our proposed model achieves better performance than the state-of-the-art ones.

1 Introduction

Traffic forecasting supports ITS but remains difficult for long-term, network-wide prediction because traffic dependencies are dynamic and poorly represented by fixed graph structures. Traffic Transformer adapts Transformer attention to learn hierarchical spatial-temporal relationships from data.

  • Traffic speed prediction is treated as foundational to ITS efforts to address worsening urban congestion.
  • Grid-based transformations lose road-network information, motivating graph-based modeling for physically connected traffic data.
  • GCN-based models depend on adjacent matrices that may be difficult to define and cannot represent changing traffic situations reliably.
  • Traffic and language data share sequential structure, long-range dependencies, positional effects, and context-dependent relationships.
  • Traffic Transformer adapts Transformer by dynamically extracting node relationships, hierarchically stacking features, and fusing global and local representations through attention.

2 Literature Review

Traffic forecasting research progressed from parametric and non-parametric methods to deep learning, grid-based models, and graph-based approaches. Transformer is presented as suitable for traffic because it models sequences and non-local relationships.

  • Prior traffic forecasting approaches are organized into traditional, grid-based deep learning, and graph-based deep learning stages according to spatial-feature extraction.
  • Traditional approaches: ARIMA and related parametric methods model traffic distributions with parameters and variants that account for periodic or additional system effects.
  • Traditional approaches: Parametric methods may perform poorly during sudden traffic changes because traffic-state distributions are complex and dynamic.
  • Traditional approaches: Non-parametric methods such as k-NN, Bayesian networks, SVMs, and ANNs were developed for prediction but often focus on short-term forecasting in regular periods.
  • Grid-based deep learning: Deep neural networks extract nonlinear spatial and temporal features using architectures including LSTM, RNN, CNN, and hybrid CNN-LSTM models.
  • Graph-based deep learning: GCNs extend convolution to graph-structured traffic data, enabling extraction of network-wide spatial features aligned with road connectivity.
  • Transformer abandons CNNs and RNNs for attention and feed-forward layers, supporting sequence and non-local modeling relevant to traffic spatial features.

3 Methodology

Traffic Transformer adapts Transformer-based attention to network-wide traffic forecasting, extracting hierarchical global and local spatial features dynamically from traffic data. Its Global Encoder and Global-Local Decoder combine these features for final prediction.

  • Traffic Transformer models network-wide traffic forecasting by learning hierarchical traffic spatiotemporal features.
  • Overall Architecture: The architecture stacks Global Encoder and Global-Local Decoder blocks to extract global and local spatial features, respectively.The decoder also fuses global and local spatial features.
  • Global Encoder: Multi-head attention learns relationships and weights between every pair of nodes, allowing global spatial features to change dynamically with input data.Queries, keys, and values are projected into multiple learned representation subspaces.
  • Global Encoder: The Global Encoder applies multi-head attention followed by a fully connected feed-forward layer, with residual connections and layer normalization around each sub-layer.The feed-forward layer uses two linear projections with ReLU activation between them.
  • Global-Local Decoder: The Global-Local Decoder uses masked multi-head attention with a K-hop adjacency mask to extract local spatial features, then fuses them with global features through attention.The encoder output supplies keys and values, while masked attention output supplies queries.
  • Temporal Embedding: A single temporal block is placed at the model beginning instead of stacking temporal and spatial blocks alternately.The paper states this design is sufficient for extracting temporal features and improves efficiency by projecting the input to a lower-dimensional representation.

4 Experiments and Results

Experiments evaluate Traffic Transformer on three traffic datasets against traditional, recurrent, and GCN-based baselines using multiple forecasting horizons and accuracy measures. Results show strong performance, particularly for long-term prediction, while attention analyses examine dynamic and hierarchical spatial relationships.

  • Data and evaluation: Experiments use METR-LA, Urban-BJ, and Ring-BJ, with 12-step inputs and outputs representing 60-minute observation and prediction horizons.METR-LA contains 207 sensors; Urban-BJ and Ring-BJ contain 278 and 236 nodes, respectively.
  • Data and evaluation: Models are evaluated with MAE, MAPE, and RMSE against ARIMA, FC-LSTM, DCRNN, STGCN, and GWN.The metrics assess overall accuracy, low-speed congestion tracking, and prediction bias and variance.
  • Model comparisons: Traffic Transformer and Traffic Transformer Encoder achieve the best performance across datasets and prediction terms, except GWN’s lower short- and middle-term RMSE on Ring-BJ.The reported advantage is especially pronounced for 60-minute forecasting, where hidden long-distance spatial relationships are beneficial.
  • Model comparisons: Learnable spatial relationships outperform fixed alternatives, while Traffic Transformer derives input-dependent relationships through multi-head attention and hierarchical feature extraction.Unlike GWN’s once-per-iteration adaptive adjacency, the Transformer’s learned relationships vary with input data.
  • Model interpretation: Attention heatmaps show that influential source relationships differ across time periods and layers, supporting dynamic and hierarchical extraction of global and local spatial features.Layer 1 and layer 6 favor global features, whereas layers 2 and 4 favor local features.

5 Conclusions and Discussions

Traffic Transformer achieves state-of-the-art traffic forecasting while dynamically learning hierarchical spatial dependencies and influential sensors. The conclusions also identify scope boundaries involving network complexity, external information, and sensor count.

  • 5 Conclusions and Discussions: Traffic Transformer achieves state-of-the-art results across three real-world datasets and identifies hierarchical influential network nodes.The model is evaluated against eight benchmark models using MAE, MAPE, and RMSE.
  • 5 Conclusions and Discussions: The model extracts dynamic and hierarchical spatiotemporal features from data rather than relying only on fixed predefined adjacency matrices.Different time periods can have different influential sensors, while spatial dependencies may be global, local, and hierarchical.
  • 5 Conclusions and Discussions: As network complexity increases, human-defined adjacency matrices become harder to specify reliably, making learned spatial dependencies preferable in complex networks.For moderately complex networks, predefined adjacency can still help the model represent local structure and improve performance.
  • 5 Conclusions and Discussions: The study is limited to speed forecasting and networks with fewer than 300 sensors because Transformer complexity is O(n^2).Future extensions include additional external information, other traffic states and transportation systems, and Transformer variants for larger networks.
Loading 2104.05163v1…