Source-linked AI summary
Spatio-Temporal Graph Transformer Networks for Pedestrian Trajectory Prediction
Cunjun Yu, Xiao Ma, Jiawei Ren, Haiyu Zhao, Shuai Yi
TL;DR
Crowd trajectory prediction requires modeling socially aware spatial interactions together with complex temporal dependencies. STAR uses attention-only spatial and temporal Transformers, including TGConv and external graph memory, and achieves state-of-the-art performance on five real-world datasets. The authors note that past trajectories alone can miss unpredictable sharp turns.
Problem
Crowd trajectory prediction must capture multimodal human interactions and their coupled spatial-temporal dependencies for applications such as surveillance and autonomous driving.
Method
STAR models crowd interactions with TGConv and interleaved spatial and temporal Transformers, while an external read-writable memory module supports temporal prediction.
Results
STAR achieves state-of-the-art performance using only attention mechanisms on five commonly used real-world pedestrian trajectory prediction datasets.
Takeaways & Limitations
STAR demonstrates that attention-only modeling can address spatio-temporal crowd trajectory prediction across five commonly used datasets.
Takeaways & Limitations
Using only past trajectories can fail to predict unpredictable sharp turns, motivating environmental information such as map or configuration data.
Abstract
from arXiv · showhide
Understanding crowd motion dynamics is critical to real-world applications, e.g., surveillance systems and autonomous driving. This is challenging because it requires effectively modeling the socially aware crowd spatial interaction and complex temporal dependencies. We believe attention is the most important factor for trajectory prediction. In this paper, we present STAR, a Spatio-Temporal grAph tRansformer framework, which tackles trajectory prediction by only attention mechanisms. STAR models intra-graph crowd interaction by TGConv, a novel Transformer-based graph convolution mechanism. The inter-graph temporal dependencies are modeled by separate temporal Transformers. STAR captures complex spatio-temporal interactions by interleaving between spatial and temporal Transformers. To calibrate the temporal prediction for the long-lasting effect of disappeared pedestrians, we introduce a read-writable external memory module, consistently being updated by the temporal Transformer. We show that with only attention mechanism, STAR achieves state-of-the-art performance on 5 commonly used real-world pedestrian prediction datasets.
1 Introduction
Crowd trajectory prediction must capture multimodal human interactions and coupled spatial-temporal dependencies. STAR addresses these challenges with attention-based graph and temporal Transformers and reports state-of-the-art performance across five real-world datasets.
- Crowd trajectory prediction is important for computer vision and robotics but must model multimodal human interactions and coupled spatial-temporal dependencies.
- Handcrafted interaction models require substantial feature engineering and often fail to represent interactions in crowded spaces.
- Attention mechanisms improve interaction modeling by assigning unequal learned importance to neighboring pedestrians instead of treating them equally.
- STAR applies self-attention to graph-based spatial interactions and temporal dependencies, interleaving spatial and temporal Transformers.Its TGConv mechanism is a Transformer-based graph convolution designed to capture complex social interactions.
- STAR achieves state-of-the-art performance on five commonly used real-world pedestrian trajectory prediction datasets using only attention mechanisms.The paper also reports extensive ablation studies of its proposed components.
2 Background
Transformers replace recurrence with self-attention for long-range sequence modeling, while graph neural networks support structured interactions through message passing. STAR extends Transformer modeling to graph sequences for crowd trajectory prediction and related graph-sequence tasks.
- Transformers replace recurrence with multi-head self-attention, jointly attending to information from different representations and positions.
- The Transformer computes query, key, and value representations, merges multiple attention-head outputs, and applies positional encoding and skip-connected updates.
- Self-attention models dependencies across time steps and can capture long-horizon temporal relationships beyond an RNN’s single-vector memory.
- Existing Transformer-based models primarily target non-structured sequences, motivating STAR’s extension to structured graph data.
- Graph neural networks process graph-structured data, while graph attention networks perform weighted message passing between nodes.
- RNN-based pedestrian predictors use sequence-to-sequence modeling with additions such as social pooling and attention mechanisms.
- STAR applies Transformer modeling to graph sequences by representing crowd interaction as a graph for trajectory prediction.The framework may also extend to event prediction in social networks and physical-system modeling, left for future study.
3 Method
STAR predicts pedestrian trajectories by interleaving temporal and spatial Transformers, using attention throughout to model individual motion, graph-based social interaction, and coupled spatio-temporal dependencies. TGConv performs Transformer-based graph message passing, while external graph memory supports temporal consistency.
- Temporal Transformer: The temporal Transformer models each pedestrian independently from trajectory embeddings and extracts temporal dependencies with self-attention.Its computation can be parallelized across pedestrians, and the paper reports better temporal modeling than RNNs in experiments.
- Temporal Transformer: The temporal Transformer architecture provides better temporal modeling than RNNs, according to the paper’s ablation studies.The temporal Transformer is described as a generalization of Transformer networks to a set of data sequences.
- Spatial Transformer: TGConv models social interaction by applying Transformer-based attention message passing over each time-step interaction graph.Pedestrians closer than distance d are connected by undirected edges, and TGConv updates each node embedding from its neighbors.
- Architecture: STAR uses two encoder modules that interleave spatial and temporal Transformers to model coupled spatio-temporal pedestrian dependencies.The first encoder extracts separate spatial and temporal information before merging features; the second further models their interaction.
- Architecture: STAR’s architecture significantly improves spatio-temporal modeling compared with naively combining spatial and temporal Transformers.Trajectory prediction uses observed pedestrian positions, and the decoder feeds the prediction at Tobs + 1 back into history for the next step.
- External Graph Memory: An external read-writable graph memory stores pedestrian embeddings and conditions current representations on previous embeddings for more consistent predictions.The memory is learnable and is updated with outputs from the second encoder module.
4 Experiments
Experiments evaluate STAR against established trajectory predictors on five pedestrian datasets, using quantitative comparisons, visualizations, and component ablations. STAR generally improves performance while modeling temporal, spatial, and spatio-temporal interactions, though sharp turns remain difficult to predict from location history alone.
- Quantitative Results: The evaluation uses ETH and UCY pedestrian datasets, compares STAR with nine predictors, and follows leave-one-out cross-validation.The benchmark includes ETH, HOTEL, ZARA1, ZARA2, and UNIV.
- Quantitative Results: STAR performs especially well across simple and complex scenes, outperforming SR-LSTM on ETH and HOTEL but remaining comparable on the denser UNIV scene.The HOTEL scene mainly contains straight-line trajectories and is relatively less crowded, whereas UNIV has high crowd density.
- Qualitative Results: STAR predicts temporally consistent trajectories and captures individual motion, social interactions, and future motions of neighboring pedestrians.Visualizations report attention to pedestrians themselves and potentially colliding neighbors, with less attention to distant or non-conflicting pedestrians.
- Limitations: STAR can still predict inaccurately when a pedestrian makes a sharp turn, because location history alone may not determine the future trajectory.The authors suggest incorporating environmental information or maps in future work.
- Ablation Studies: The temporal Transformer outperforms LSTM, TGConv outperforms alternative graph convolutions, and interleaved spatial-temporal Transformers improve spatio-temporal correlation modeling.These findings come from ablations comparing temporal encoders, spatial encoders, and single- versus two-encoder structures.
- Ablation Studies: Graph memory improves performance on all datasets and produces smoother temporal embeddings, with the largest improvement reported on ZARA1.The authors associate this smoothing with improved overall temporal modeling for STAR.
5 Conclusion
STAR uses spatial and temporal Transformers with TGConv to perform crowd trajectory prediction using only attention mechanisms, achieving SOTA performance on five commonly used datasets. The framework may need additional environmental information for unpredictable sharp turns and could extend to other graph-learning tasks.
- STAR combines spatial and temporal Transformers with TGConv in an attention-only framework for spatio-temporal crowd trajectory prediction.
- STAR may fail on unpredictable sharp turns because it predicts using only past trajectories.
- STAR and TGConv could be applied to other graph learning tasks beyond trajectory prediction.
Additional Attention Visualization
The spatial Transformer assigns attention to pedestrians in the interaction graph, with higher attention generally directed toward each pedestrian and its neighbors.
- Spatial attention visualizations show that pedestrians receive higher attention over themselves and their neighbors.Circle size represents attention value; larger circles indicate higher attention.
Ablation Trajectory Prediction Visualizations
Ablation trajectory visualizations compare histories, ground truth, and predictions, with STAR producing the best qualitative predictions across spatial and temporal dimensions.
- STAR produces the best qualitative predictions among the ablations, both spatially and temporally.Yellow lines denote history, red lines ground truth, and blue lines predictions.