Source-linked AI summary
Social-STGCNN: A Social Spatio-Temporal Graph Convolutional Neural Network for Human Trajectory Prediction
Abduallah Mohamed, Kun Qian, Mohamed Elhoseiny, Christian Claudel
TL;DR
Pedestrian trajectory prediction is important for applications such as autonomous driving but remains difficult because social interactions and environmental factors shape uncertain motion. Social-STGCNN models these interactions directly with a spatio-temporal graph and uses graph and temporal convolutions to predict trajectories in one pass. It outperforms prior methods across prediction accuracy, model size, inference speed, and data efficiency, including a 20% FDE improvement and state-of-the-art ADE with 20% of the training data.
Problem
Pedestrian trajectories are difficult to predict because they depend on complex interactions with the environment, other pedestrians, and unknown destinations or intended paths.
Method
Social-STGCNN represents pedestrian interactions as a spatio-temporal graph with a kernel-weighted adjacency matrix and processes it using graph and temporal CNNs.
Results
Social-STGCNN outperforms prior methods on prediction error, computational time, parameter count, and data efficiency, including 20% lower FDE than the previous state of the art.
Takeaways & Limitations
The model provides more realistic forecasts for behaviors including collision avoidance, parallel walking, and meeting a group, while learning from few training samples.
Takeaways & Limitations
Earlier recurrent and aggregation-based approaches have limitations, including parameter inefficiency, expensive training, unintuitive interaction modeling, and potentially leaky pooling.
Abstract
from arXiv · showhide
Better machine understanding of pedestrian behaviors enables faster progress in modeling interactions between agents such as autonomous vehicles and humans. Pedestrian trajectories are not only influenced by the pedestrian itself but also by interaction with surrounding objects. Previous methods modeled these interactions by using a variety of aggregation methods that integrate different learned pedestrians states. We propose the Social Spatio-Temporal Graph Convolutional Neural Network (Social-STGCNN), which substitutes the need of aggregation methods by modeling the interactions as a graph. Our results show an improvement over the state of art by 20% on the Final Displacement Error (FDE) and an improvement on the Average Displacement Error (ADE) with 8.5 times less parameters and up to 48 times faster inference speed than previously reported methods. In addition, our model is data efficient, and exceeds previous state of the art on the ADE metric with only 20% of the training data. We propose a kernel function to embed the social interactions between pedestrians within the adjacency matrix. Through qualitative analysis, we show that our model inherited social behaviors that can be expected between pedestrians trajectories. Code is available at https://github.com/abduallahmohamed/Social-STGCNN.
1. Introduction
Pedestrian trajectory prediction matters for autonomous driving and surveillance but is difficult because trajectories reflect complex social, environmental, and stochastic factors. Social-STGCNN addresses limitations of recurrent architectures and aggregation layers by representing interactions directly as a spatio-temporal graph.
- Accurate pedestrian trajectory prediction supports autonomous-driving planning, collision avoidance, emergency braking, and surveillance applications.
- 70% of pedestrians tend to walk in groups, making social conventions and behaviors central to trajectory prediction.The paper highlights parallel walking, group motion, collision avoidance, and merging as challenging behaviors.
- Earlier methods commonly used recurrent models and aggregation layers to combine learned pedestrian states.Social-LSTM and related approaches interacted through pooling or other aggregation mechanisms.
- Recurrent architectures are described as parameter inefficient and expensive to train, while aggregation mechanisms may be unintuitive or fail to model interactions correctly.The paper also notes that pooling can be leaky in information.
- Social-STGCNN replaces aggregation layers with a spatio-temporal graph whose weighted adjacency matrix uses a kernel to quantify pedestrian influence.Graph and temporal CNNs process the graph and predict the whole sequence in a single shot.
2. Related work
Related work primarily models pedestrian motion with recurrent networks, pooling or attention-based interaction mechanisms, visual features, and generative approaches. Social-STGCNN instead adapts spatio-temporal graph CNNs to extract spatial and temporal information for trajectory prediction.
- Social-LSTM models each pedestrian with a recurrent network, aggregates recurrent outputs through pooling, and assumes a bi-variate Gaussian trajectory distribution.
- Later methods such as PIF and SR-LSTM extend this direction with visual features and new pooling or weighting mechanisms.Social-BiGAT similarly weights contributions through attention over recurrent trajectory states.
- Social-STGCNN adapts a CNN architecture originally designed for skeleton-based action recognition to extract spatial and temporal information from pedestrian graphs.The resulting embedding is used to predict all pedestrians’ future trajectories.
3. Problem Formulation
The problem formulation represents observed pedestrian positions over time and predicts future trajectories over a specified horizon. Predicted locations are modeled with bi-variate Gaussian distributions and trained using negative log-likelihood.
- Given N pedestrians and observed positions over a period T_o, the task is to predict their future trajectories over horizon T_p.
- Each pedestrian’s trajectory is represented as a sequence of two-dimensional locations indexed over the future time horizon.
- The location coordinates are treated as random variables describing each pedestrian’s probability distribution in 2D space.
- The model assumes each location follows a bi-variate Gaussian distribution and represents predictions through an estimated bi-variate distribution.
- Training minimizes the negative log-likelihood of the predicted distributions.
4. The Social-STGCNN Model
Social-STGCNN represents pedestrian trajectories as a spatio-temporal graph, applies graph convolutions to extract embeddings, and uses temporal convolutions to predict future trajectories. Its design replaces aggregation layers with weighted graph interactions and adds a time-extrapolation component.
- Social interaction modeling: Social-STGCNN replaces aggregation layers with graph edges and a weighted adjacency matrix whose kernel quantifies pedestrian influence.The proposed kernel function is used to represent social interactions directly in the graph.
- Model overview: The model constructs a spatio-temporal graph from observed pedestrian frames and processes it with ST-GCNNs to create an embedding.The graph contains pedestrian vertices, time-varying attributes, and weighted adjacency matrices.
- Spatio-temporal graph convolution: ST-GCNNs extend spatial graph convolution across time while preserving the graph topology and varying vertex attributes across time steps.The resulting embedding compactly represents the observed trajectory history.
- Time extrapolation: TXP-CNNs operate on the temporal dimension of the graph embedding and extrapolate future trajectories without recurrent state.They treat the embedding’s time dimension as feature channels and use residual-connected convolutional layers.
- Novelty: Compared with ST-GCNN, Social-STGCNN introduces a different graph-construction kernel and TXP-CNNs for future prediction.ST-GCNN was originally designed for classification, whereas TXP-CNNs enable prediction from graph embeddings.
- Implementation: The model normalizes each time-specific adjacency matrix before applying ST-GCNN layers with trainable parameters.Normalization adds self-connections and uses the diagonal node-degree matrix.
5. Datasets and Evaluation Metrics
The model is evaluated on ETH and UCY pedestrian-trajectory datasets using ADE and FDE. The evaluation observes 3.2 seconds of motion and predicts the following 4.8 seconds.
- Datasets: ETH and UCY provide the evaluation data, comprising five scenes: ETH, HOTEL, ZARA1, ZARA2, and UNIV.Trajectories are sampled every 0.4 seconds.
- Evaluation protocol: The evaluation observes 8 frames over 3.2 seconds and predicts 12 frames over the next 4.8 seconds.The training and testing strategy follows Social-LSTM’s dataset split procedure.
- Metrics: ADE measures average prediction performance along the trajectory, whereas FDE measures precision at the endpoint.Both metrics are used to evaluate model performance.
6. Experiments and Results Analysis
Experiments evaluate Social-STGCNN’s architecture, kernel choices, quantitative efficiency, data efficiency, and qualitative social behavior. The model achieves strong prediction performance while using a compact graph-based design and producing socially plausible trajectory samples.
- Model configuration: One ST-GCNN layer and five TXP-CNN layers provide the best reported Social-STGCNN configuration on ADE/FDE.The ablation varies ST-GCNN and TXP-CNN depth and identifies this configuration as optimal.
- Kernel function: The inverse L2-based similarity kernel is selected for the weighted adjacency matrix because it better represents stronger influence from nearby pedestrians.The kernel weights vertex contributions during graph convolution, and the reported ablation identifies the similarity function as best-performing.
- Quantitative performance: Social-STGCNN outperforms previous methods on both ADE and FDE, including an FDE of 0.75 versus SR-LSTM’s 0.94.The reported FDE is about 20% lower than the previous state of the art, while ADE is 2% better than SR-LSTM.
- Efficiency: 7.6K parameters and 0.002 seconds per inference step make Social-STGCNN substantially smaller and faster than prior models.The model is reported as about one sixth the size of S-GAN-P and about 48× faster in inference.
- Data efficiency: With 20% of the training data, Social-STGCNN exceeds state of the art on FDE and surpasses Social-GAN on ADE.The comparison uses randomly selected subsets shared across models; the paper attributes the model’s data efficiency to parameter efficiency.
- Qualitative analysis: Qualitative samples capture collision avoidance, speed changes, direction changes, and parallel walking, while some samples still show collisions or divergence.The qualitative analyses compare kernel variants and Social-GAN across pedestrian interaction scenarios.
7. Conclusion
Social-STGCNN improves pedestrian trajectory prediction across prediction error, computational efficiency, parameter count, and data efficiency. Qualitative analyses also report more realistic forecasts in social-interaction scenarios.
- Social-STGCNN improves prediction error, computational time, and parameter count over previous methods.
- The model is data-efficient and can learn from few training samples.
- Social-STGCNN provides more realistic forecasts for collision avoidance, parallel walking, and individual-meets-group scenarios.
- 8.5 times fewer parameters and up to 48 × faster inference are reported compared with previous models.
Supplementary: More Qualitative Results
Figure 6 compares averaged trajectory predictions on ETH scenes, showing that Social-STGCNN generally tracks ground truth more precisely than linear and Social-GAN predictions, while also exhibiting failures.
- Figure 6 compares averaged trajectory predictions from different models on scenes from the ETH dataset.
- Social-STGCNN tracks and aligns more precisely with ground truth than linear and Social-GAN predictions in the first four rows.The competing predictions diverge and do not account for variation in pedestrian motion.
- The last row contains cases where Social-STGCNN's averaged trajectory fails.