Source-linked AI summary
PDFormer: Propagation Delay-Aware Dynamic Long-Range Transformer for Traffic Flow Prediction
Jiawei Jiang, Chengkai Han, Wayne Xin Zhao, Jingyuan Wang
TL;DR
Traffic flow prediction requires modeling complex spatial-temporal dependencies, but existing GNN-based methods often miss dynamic, long-range, and delayed spatial interactions. PDFormer combines spatial-temporal self-attention with geographic and semantic masking and delay-aware transformation; experiments on six public datasets report state-of-the-art performance, competitive efficiency, and interpretable attention.
Problem
Traffic flow prediction needs to model complex spatial-temporal dependencies, while existing GNN methods commonly overlook dynamic spatial relations, long-range dependencies, and propagation delays.
Method
PDFormer uses spatial-temporal self-attention with geographic and semantic graph masks plus a delay-aware feature transformation to model dynamic, long-range, and delayed dependencies.
Results
Experiments on six real-world public traffic datasets report state-of-the-art performance, competitive computational efficiency, and highly interpretable learned spatial-temporal attention.
Takeaways & Limitations
PDFormer provides a traffic flow prediction model that addresses dynamic, long-range, and time-delayed spatial-temporal characteristics while exposing learned attention for interpretation.
Abstract
from arXiv · showhide
As a core technology of Intelligent Transportation System, traffic flow prediction has a wide range of applications. The fundamental challenge in traffic flow prediction is to effectively model the complex spatial-temporal dependencies in traffic data. Spatial-temporal Graph Neural Network (GNN) models have emerged as one of the most promising methods to solve this problem. However, GNN-based models have three major limitations for traffic prediction: i) Most methods model spatial dependencies in a static manner, which limits the ability to learn dynamic urban traffic patterns; ii) Most methods only consider short-range spatial information and are unable to capture long-range spatial dependencies; iii) These methods ignore the fact that the propagation of traffic conditions between locations has a time delay in traffic systems. To this end, we propose a novel Propagation Delay-aware dynamic long-range transFormer, namely PDFormer, for accurate traffic flow prediction. Specifically, we design a spatial self-attention module to capture the dynamic spatial dependencies. Then, two graph masking matrices are introduced to highlight spatial dependencies from short- and long-range views. Moreover, a traffic delay-aware feature transformation module is proposed to empower PDFormer with the capability of explicitly modeling the time delay of spatial information propagation. Extensive experimental results on six real-world public traffic datasets show that our method can not only achieve state-of-the-art performance but also exhibit competitive computational efficiency. Moreover, we visualize the learned spatial-temporal attention map to make our model highly interpretable.
Introduction
Traffic flow prediction must capture dynamic, long-range, and delayed spatial-temporal dependencies. PDFormer addresses these limitations with dynamic spatial self-attention, short- and long-range graph masks, delay-aware transformation, and temporal attention.
- Accurate traffic flow prediction supports route planning, vehicle dispatching, and congestion relief.
- GNN-based traffic prediction methods commonly model spatial dependencies statically, limiting their ability to learn time-varying urban traffic patterns.Traffic correlations can strengthen during peak periods and weaken at other times.
- Existing methods often miss long-range spatial dependencies between distant locations with similar traffic patterns, partly because GNNs suffer from over-smoothing.
- Traffic conditions can propagate between locations with a time delay, such as when an accident takes several minutes to affect another location.
- PDFormer combines spatial self-attention, geographic and semantic graph masking, delay-aware feature transformation, and temporal self-attention for traffic flow prediction.The design targets dynamic, short- and long-range, and delayed spatial-temporal dependencies.
- Experiments on six real-world public datasets report state-of-the-art performance, competitive computational efficiency, and interpretable learned spatial-temporal attention.
PRELIMINARIES
The paper formalizes traffic flow prediction on a graph-structured road network and describes PDFormer as a framework for mapping historical observations to future traffic flows.
- Notations and Definitions: A road network is represented as a graph G = (V, E, A), with nodes, edges, and an adjacency matrix.N denotes the number of nodes in the graph.
- Notations and Definitions: Traffic flow at time t is represented as Xt ∈RN×C, while X ∈RT×N×C contains observations across T time slices.C is the traffic-flow dimension, such as inflow and outflow when C = 2.
- Traffic Flow Prediction: Traffic flow prediction learns a mapping from previous T steps of observations to future T′ steps of traffic flow.
Methods
PDFormer combines dynamic spatial-temporal self-attention with geographic and semantic graph masking, delay-aware feature transformation, and heterogeneous attention fusion. Its embedding layer incorporates road-network, periodic, and positional information before spatial-temporal encoding.
- Data Embedding Layer: The data embedding layer projects raw inputs into high-dimensional representations and sums spatial, weekly, daily, and temporal position embeddings.Spatial embeddings encode road-network structure through graph Laplacian eigenvectors, while temporal embeddings represent weekly and daily traffic periodicity.
- Spatial-Temporal Encoder Layer: PDFormer’s encoder combines geographic, semantic, and temporal self-attention to model dynamic spatial-temporal dependencies.Geographic and semantic modules capture short- and long-range spatial dependencies, while temporal attention models long-range temporal patterns.
- Spatial-Temporal Encoder Layer: Spatial self-attention uses geographic and semantic graph masks to retain nearby node interactions and connect distant nodes with similar traffic patterns.The geographic mask thresholds graph hops, while the semantic mask uses Dynamic Time Warping similarity to select relevant long-range nodes.
- Spatial-Temporal Encoder Layer: Delay-aware feature transformation derives delay information from each node’s short-term historical traffic patterns and inserts it into geographic attention keys.Representative patterns are extracted with sliding windows and k-Shape clustering, then fused with node histories; this models delayed spatial propagation.
- Spatial-Temporal Encoder Layer: Heterogeneous attention heads concatenate geographic, semantic, and temporal outputs before projection, integrating spatial and temporal information in one block.Skip connections after encoder layers convert outputs to a common skip dimension and sum them for the final hidden state.
Experiments
PDFormer is evaluated across six public traffic datasets against 17 baselines, with experiments covering predictive performance, ablations, interpretability, and computational efficiency. It consistently outperforms baselines, benefits from its spatial masks and delay-aware module, and maintains competitive efficiency.
- Experimental Setup: PDFormer is evaluated on six real-world public datasets against 17 baselines spanning grid-based, time-series, GNN-based, and self-attention models.The datasets include PeMS04, PeMS07, PeMS08, NYCTaxi, CHIBike, and T-Drive.
- Experimental Setup: The experiments use different forecasting setups for graph-based and grid-based datasets, including multi-step and single-step prediction.Graph-based datasets use 12 historical steps to predict the next 12, while grid-based datasets use six steps to predict the next one.
- Performance Comparison: PDFormer significantly outperforms all baselines across all metrics and datasets, improving over the second-best method by 4.58% in MAE, 5.00% in MAPE, and 4.79% in RMSE.The reported superiority is supported by Student’s t-test at level 0.01.
- Ablation Study: Ablations show that spatial self-attention, both geographic and semantic masks, and delay-aware transformation each contribute to performance.Removing masks, either spatial dependency type, or delay modeling produces worse results than PDFormer.
- Case Study: Attention visualizations show that the spatial masks help PDFormer combine nearby dependencies with distant locations sharing similar traffic patterns or urban functions.The masked model avoids diffuse attention and identifies relevant functional areas beyond immediate neighborhoods.
- Model Efficiency Study: PDFormer achieves competitive computational efficiency and reduces PeMS04 training and inference time by over 35% and 80%, respectively, compared with ASTGNN.The comparison reports average training and inference time per epoch.
Related Work
Traffic prediction research progressed from CNNs and GNNs toward attention-based architectures, including Transformers, while computational comparisons are reported for self-attention models.
- CNNs were initially applied to grid-based traffic data, followed by widespread use of GNNs for traffic prediction.
- Training and inference time per epoch are compared among self-attention-based models in seconds.
- Transformers use self-attention mechanisms and have demonstrated effectiveness across language, vision, and representation-learning tasks.
Conclusion
PDFormer combines spatial-temporal self-attention with delay-aware feature transformation for traffic flow prediction. Experiments on six real-world datasets support its reported superiority, while visualization improves interpretability.
- PDFormer uses spatial-temporal self-attention to capture dynamic spatial and temporal traffic patterns.
- A delay-aware feature transformation module explicitly models time delay in spatial information propagation.
- Experiments on six real-world datasets demonstrate the reported superiority of PDFormer.
- The learned attention map is visualized to make the model interpretable.
- Future work includes applying PDFormer to wind power forecasting and exploring pre-training to address insufficient data.