Source-linked AI summary
Social-BiGAT: Multimodal Trajectory Forecasting using Bicycle-GAN and Graph Attention Networks
Vineet Kosaraju, Amir Sadeghian, Roberto Martín-Martín, Ian Reid, S. Hamid Rezatofighi, Silvio Savarese
TL;DR
Pedestrian forecasting must model social interactions, scene constraints, and multiple plausible future trajectories. Social-BiGAT combines graph attention, adversarial recurrent prediction, and a reversible scene–latent mapping, and reports state-of-the-art performance across trajectory benchmarks.
Problem
Existing trajectory forecasting methods address social or physical cues but largely ignore the multimodal nature of each pedestrian’s future trajectory.
Method
Social-BiGAT uses a graph attention network with an adversarial recurrent encoder-decoder and a Bicycle-GAN-inspired reversible mapping between scenes, trajectories, and latent noise.
Results
Social-BiGAT achieves the best performance among the proposed architectures and outperforms prior state-of-the-art methods across widely used trajectory benchmarks.
Takeaways & Limitations
The model generates multimodal trajectories for multiple pedestrians, captures social behavior, and allows predictions to be controlled by changing latent variables at test time.
Abstract
from arXiv · showhide
Predicting the future trajectories of multiple interacting agents in a scene has become an increasingly important problem for many different applications ranging from control of autonomous vehicles and social robots to security and surveillance. This problem is compounded by the presence of social interactions between humans and their physical interactions with the scene. While the existing literature has explored some of these cues, they mainly ignored the multimodal nature of each human's future trajectory. In this paper, we present Social-BiGAT, a graph-based generative adversarial network that generates realistic, multimodal trajectory predictions by better modelling the social interactions of pedestrians in a scene. Our method is based on a graph attention network (GAT) that learns reliable feature representations that encode the social interactions between humans in the scene, and a recurrent encoder-decoder architecture that is trained adversarially to predict, based on the features, the humans' paths. We explicitly account for the multimodal nature of the prediction problem by forming a reversible transformation between each scene and its latent noise vector, as in Bicycle-GAN. We show that our framework achieves state-of-the-art performance comparing it to several baselines on existing trajectory forecasting benchmarks.
1 Introduction
Pedestrian trajectory forecasting must account for social interactions, physical scene context, and multiple plausible futures. Social-BiGAT addresses these challenges with graph attention, reversible trajectory–latent mappings, and scene cues.
- Trajectory forecasting supports collision avoidance, crowd management, tracking, and re-identification across autonomous vehicles, social robotics, and surveillance applications.
- Pedestrian behavior depends on social interactions, physical scene constraints, and multimodal choices with variation within each behavioral mode.Examples include collision avoidance, group walking, and choosing left or right when pedestrians approach one another.
- Handcrafted social rules can generalize poorly, recurrent models may omit physical scene cues, and prior GANs fail to learn truly multimodal behavior.Existing GANs either use one social vector for all pedestrians or rely on hand-defined sorting that may not work optimally in every scene.
- Social-BiGAT uses a fully interactive graph attention network to model pedestrians, a reversible trajectory–latent mapping to learn multimodality, and soft attention for physical scene cues.The design aims to generate socially and physically acceptable trajectories while learning a broader multimodal distribution from single samples across scenes.
2 Related Work
Related work spans handcrafted motion rules, recurrent models, interaction mechanisms, generative forecasting, and scene-aware approaches. Social-BiGAT’s graph formulation builds on graph attention’s ability to model weighted global interactions without hand-defined pooling or sorting.
- Handcrafted trajectory rules and energy parameters often fail to generalize, motivating recurrent networks that learn motion parameters from data.
- Prior interaction models may restrict attention to nearby pedestrians, omit global interactions, or struggle with variable numbers of humans.
- Graph attention networks apply self-attention to graph-structured data and implicitly assign different importance to nodes through edge weights.
- Modeling pedestrians as nodes in a fully connected graph enables efficient local and global interaction modeling without pooling or sorting systems that may lose features.
- Image translation research progressed from paired pix2pix mappings to unpaired CycleGAN mappings using cycle consistency, motivating multimodal generative modeling.
3 Social-BiGAT
Social-BiGAT forecasts pedestrians’ future trajectories from observed movements and scene context using a multimodal adversarial architecture. Its GAT models fully connected social interactions, while a latent encoder creates a reversible mapping between scene trajectories and noise.
- 3.1 Problem Definition: The task predicts future 2D pedestrian coordinates from prior movements and contextual scene information for multiple visible pedestrians.The model aims to generate feasible samples from the potentially multimodal distribution of future trajectories.
- 3.2 Overall Model: Social-BiGAT comprises a generator, local and global discriminators, and a latent-space encoder.The generator includes feature encoding, attention, and decoding modules; the discriminators operate at pedestrian and scene scales.
- 3.3 Feature Encoder: The feature encoder combines LSTM representations of observed pedestrian displacements with CNN representations of the physical scene.These social and physical features provide inputs to the model’s attention and generation components.
- 3.4 Attention Network: Graph attention layers assign interaction weights among pedestrians, avoiding identical pooled features and hand-defined distance ordering.The pedestrian graph remains fully connected, allowing every pedestrian to interact without imposing an order restriction.
- 3.5 GAN Network: A decoder LSTM generates trajectories from pedestrian, social, physical, and noise features, while discriminators assess local and global realism.The generator is conditioned on a noise vector and scene-derived contexts; the global discriminator evaluates the combined scene context.
- 3.6 Latent Encoder: The latent encoder promotes multimodality by mapping generated trajectories back to their originating noise and learning scene-level latent statistics.Training performs both noise-to-trajectory-to-noise and trajectory-to-noise-to-trajectory transformations.
4 Experiments
Experiments on ETH and UCY evaluate Social-BiGAT against deterministic and generative baselines using ADE and FDE. The combined GAT and BiGAN architecture achieves the strongest reported performance, generalizes better with fewer samples, and produces varied, socially plausible trajectories.
- Quantitative Results: Social-BiGAT is evaluated on ETH and UCY across five scenes, using ADE and FDE for 12 future timesteps.The evaluation compares Linear, S-LSTM, S-GAN-P, Sophie, GAT, BiGAN, and Social-BiGAT; generative models use K = 20 samples in Table 1.
- Quantitative Results: 0.15 meters: Social-BiGAT decreases average FDE from the previous state-of-the-art model and achieves the best performance among the evaluated architectures.The authors attribute this improvement to reduced errors in the Hotel scene compared with other generative architectures.
- Quantitative Results: GAT improves performance alone, whereas BiGAN alone does not help at K = 20; combining both components yields the strongest model.The comparison isolates the graph attention network and latent scene encoder before evaluating the complete Social-BiGAT architecture.
- Quantitative Results: At lower K, Social-BiGAT’s ADE and FDE increase more slowly than S-GAN-P and Sophie’s, indicating better generalization with fewer generated samples.The authors associate the slower increase with reduced variance in the output trajectory distributions from the latent scene encoder.
- Qualitative Results: Across four scenes, Social-BiGAT shows lower variance, better crowd and group interaction modeling, and realistic collision-avoidance trajectories than S-GAN-P and Sophie.Figure 4 compares observed trajectories, ground-truth futures, and generated samples across the three models.
- Qualitative Results: Varying latent z produces interpretable behavioral modes, including avoidance versus aggressiveness, linearity versus curvature, and fast versus slow movement.The visualized changes support the use of a Bicycle-GAN-inspired architecture for controllable trajectory generation.
5 Conclusion
The conclusion presents Social-BiGAT as a multimodal trajectory forecaster that models interactions among multiple pedestrians and allows test-time control through latent variables. Its GAT and local-global discriminator design produces more realistic human-motion predictions across trajectory benchmarks.
- Social-BiGAT outperforms prior state-of-the-art methods across several widely used trajectory benchmarks.
- The model generates multiple multimodal trajectories for multiple humans and controls predictions by adjusting latent variables at test time.
- Social-BiGAT uses a social attention graph network and two discriminators operating at local and global scales.
- These design patterns generate pedestrian trajectories that more realistically predict human motion.