Source-linked AI summary
Recursive Social Behavior Graph for Trajectory Prediction
Jianhua Sun, Qinhong Jiang, Cewu Lu
TL;DR
Human trajectory forecasting needs better modeling of social interaction because distance-based approaches can struggle with complex pedestrian relationships. The paper introduces a group-supervised Recursive Social Behavior Graph with recursive social representations and GCN propagation, achieving 11.1% ADE and 10.8% FDE improvement over a state-of-the-art method. The reported failure on UNIV is associated with testing on large pedestrian groups absent from leave-one-out training.
Problem
Distance-based interaction methods can fail in complicated scenes, while social relationships are too complex to capture reliably with handcrafted representations.
Method
RSBG recursively extracts group-supervised social representations into a graph and uses GCNs to propagate interaction information for trajectory prediction.
Results
11.1% improvement in ADE and 10.8% in FDE are reported against a state-of-the-art method across multiple human-trajectory datasets.
Takeaways & Limitations
Group-based social interaction modeling helps the model learn latent social relations and perform better than distance-based methods.
Takeaways & Limitations
On UNIV, performance degradation may result from leave-one-out training that excludes huge groups while evaluation includes scenes with 20 or more pedestrians.
Abstract
from arXiv · showhide
Social interaction is an important topic in human trajectory prediction to generate plausible paths. In this paper, we present a novel insight of group-based social interaction model to explore relationships among pedestrians. We recursively extract social representations supervised by group-based annotations and formulate them into a social behavior graph, called Recursive Social Behavior Graph. Our recursive mechanism explores the representation power largely. Graph Convolutional Neural Network then is used to propagate social interaction information in such a graph. With the guidance of Recursive Social Behavior Graph, we surpass state-of-the-art method on ETH and UCY dataset for 11.1% in ADE and 10.8% in FDE in average, and successfully predict complex social behaviors.
1. Introduction
Human trajectory forecasting must model human-human interaction to generate plausible paths, but distance-based methods struggle with complicated relationships. The paper introduces group-supervised Recursive Social Behavior Graphs and reports improved ADE and FDE over the state of the art.
- Trajectory forecasting supports computer vision, autonomous driving, and human-robot interaction, but plausible paths require modeling human-human interaction.
- Distance-based force, pooling, and attention methods can fail when interactions are complicated because they use spatial distance to estimate influence.
- Groups capture shared movements, behaviours, purposes, or destinations, while their relationships can be represented as a directional graph.
- RSBG recursively extracts social relationships, supervises them with group annotations, and uses GCNs to propagate interaction information.
- 11.1% improvement in ADE and 10.8% in FDE are reported against a state-of-the-art method across several video datasets.
2. Related Work
Prior trajectory-forecasting methods range from mathematical models and recurrent networks to pooling, attention, and graph networks. Their treatment of social interaction often remains constrained by handcrafted assumptions or spatial distance.
- Human trajectory forecasting: Early trajectory-forecasting methods formulate prediction with mathematical models, including shortest-path optimization and sequential decision-making.
- Human trajectory forecasting: Deep-learning approaches use pooling, GANs, and contextual features to propagate interactions, model multimodality, or improve prediction accuracy.
- Human-human interactions in trajectory forecasting: Force-based interaction models depend on prior knowledge and may not handle sophisticated crowds with pedestrians exhibiting different behaviors.
- Human-human interactions in trajectory forecasting: Social-LSTM and related pooling methods integrate social features but use inter-person distance to determine relationship strength.
- Human-human interactions in trajectory forecasting: Attention-based methods also rely on distance, while GAT removes that restriction but lacks supervision and may fail in sophisticated scenes.
- Graph Neural Network: GCNs are graph-neural-network variants widely used in computer vision tasks, providing a relevant framework for graph-based feature propagation.
3. Approach
The approach models pedestrian interactions from group-based social relationships rather than distance alone, recursively builds an RSBG, and combines propagated social features with individual representations for trajectory decoding.
- Input and individual representation: Trajectory forecasting uses observed coordinates and surrounding image patches to predict future coordinate sequences.The input covers [1, Tobs], while the output covers [Tobs+1, Tobs+pred].
- Input and individual representation: BiLSTMs encode historical trajectories, CNNs encode human context, and their outputs are concatenated into each pedestrian’s individual representation.The BiLSTM captures dependencies in both previous and future observed steps, while CNNs process trajectory-centered image patches.
- Relational social representation: Group annotations supervise learned relational representations, avoiding reliance on pedestrian distance as the sole relationship cue.Experts annotate dynamic groups using physical and sociological information, with 0/1 labels indicating whether pedestrians share a group.
- Recursive Social Behavior Graph: The recursive mechanism updates individual and relational features across depths, then averages R0, R1, and R2 into Ra for graph construction.Ra is trained against ground-truth relationships with cross-entropy loss and serves as the adjacency matrix for graph convolution.
- Recursive Social Behavior Graph: RSBG represents pedestrians as graph vertices and relational social representations as directed edges, providing features for trajectory generation.The graph is formed from individual trajectory features and the recursively generated relational matrix Ra.
- Social propagation and decoding: A GCN propagates social information between adjacent nodes, after which social and individual features are concatenated and decoded by an LSTM.The model introduces GCN message passing to integrate social interaction information guided by RSBG.
- Trajectory generation: The Exponential L2 Loss is proposed because conventional L2 loss does not emphasize final displacement error sufficiently.The proposed loss multiplies the L2 objective by a coefficient that grows over time.
4. Experiments
Experiments evaluate RSBG on ETH and UCY using ADE/FDE, benchmark comparisons, ablations, and qualitative social-behavior cases. The model improves average performance over STGAT while revealing limitations on UNIV and with context features in some subsets.
- Evaluation setup: Evaluation uses ETH and UCY pedestrian cases, observing 3.2 seconds and predicting 4.8 seconds with leave-one-out training and evaluation.The benchmarks include ETH, HOTEL, UNIV, ZARA1, and ZARA2.
- Evaluation metrics: ADE measures average trajectory distance, while FDE measures destination error at the final prediction timestep.
- Quantitative results: 11.1% ADE and 10.8% FDE improvements are achieved on average over STGAT across the ETH and UCY benchmark.The comparison uses Tpred = 12 and reports stronger performance on most subsets.
- Quantitative results: The model fails against STGAT on UNIV, possibly because leave-one-out evaluation tests large pedestrian groups absent from the corresponding training data.UNIV contains scenes with 20 or more pedestrians, unlike the other datasets.
- Ablation study: BiLSTM encoders improve average ADE by 5.9% and FDE by 4.8% compared with LSTM encoders.The ablation changes only the historical-trajectory encoder while retaining other modules.
- Ablation study: Exponential L2 loss with γ = 20 reduces average ADE by 4.0% and FDE by 4.8%, whereas excessive FDE emphasis harms performance.The loss addresses the limited emphasis that standard L2 loss places on final-position accuracy.
- Qualitative analysis: Qualitative cases show RSBG predicting joining, following, and collision avoidance behaviors that competing methods sometimes miss or misdirect.The visualizations compare RSBG with STGAT(1V-1) and SGAN(1V-1).
- Qualitative analysis: RSBG assigns strong relational weights to behaviorally linked pedestrians, including distant pedestrians involved in potential conflicts.The visualized relationships are reported to remain informative beyond simple distance-based interaction.
5. Conclusion
The paper models human-human interactions through a Recursive Social Behavior Graph supervised by group-based annotations. GCNs propagate social information, while an Exponential L2 Loss emphasizes final displacement error and supports improved trajectory prediction over distance-based methods.
- The paper studies human-human interactions among pedestrians to improve trajectory prediction.
- Recursive Social Behavior Graph uses group-based annotations to learn latent social relations beyond spatial distance.
- GCNs integrate information from RSBG nodes and edges, while Exponential L2 Loss emphasizes final displacement error.