Source-linked AI summary
SocialVAE: Human Trajectory Prediction using Timewise Latents
Pei Xu, Jean-Bernard Hayet, Ioannis Karamouzas
TL;DR
Human trajectory prediction requires modeling uncertain, multimodal navigation and interpersonal variation beyond deterministic approaches. SocialVAE combines a timewise VAE with social attention and backward posterior approximation, and reports improved or state-of-the-art performance across several benchmarks. Its optional Final Position Clustering procedure further addresses sampling diversity when predictions are limited.
Problem
Human navigation is multimodal and uncertain, with multiple possible trajectories and behavior differences that deterministic and rule-based methods do not effectively capture.
Method
SocialVAE uses a timewise VAE with per-step stochastic latent variables, backward posterior approximation, and attention over neighboring agents’ social features.
Results
SocialVAE reports state-of-the-art performance on ETH/UCY and SDD, captures multimodal behavior in everyday and NBA scenarios, and achieves more than 10% improvement overall, with some cases exceeding 50%.
Takeaways & Limitations
Final Position Clustering can reduce sampling bias and improve prediction diversity and quality when only a limited number of trajectory samples is drawn.
Abstract
from arXiv · showhide
Predicting pedestrian movement is critical for human behavior analysis and also for safe and efficient human-agent interactions. However, despite significant advancements, it is still challenging for existing approaches to capture the uncertainty and multimodality of human navigation decision making. In this paper, we propose SocialVAE, a novel approach for human trajectory prediction. The core of SocialVAE is a timewise variational autoencoder architecture that exploits stochastic recurrent neural networks to perform prediction, combined with a social attention mechanism and a backward posterior approximation to allow for better extraction of pedestrian navigation strategies. We show that SocialVAE improves current state-of-the-art performance on several pedestrian trajectory prediction benchmarks, including the ETH/UCY benchmark, Stanford Drone Dataset, and SportVU NBA movement dataset. Code is available at: https://github.com/xupei0610/SocialVAE.
1 Introduction
Human trajectory prediction must handle multimodal, uncertain navigation behavior and differences across people, which deterministic and rule-based methods model poorly. SocialVAE addresses this with timewise stochastic modeling, social attention, and backward posterior inference, achieving reported gains across trajectory benchmarks.
- Human navigation can produce multiple trajectories in the same scenario, while individual behavior varies with crowd density, lighting, weather, social context, and personality.
- SocialVAE uses a timewise VAE whose latent variables condition RNN hidden dynamics at each time step to model changing human decisions.
- A backward RNN approximates the posterior from whole trajectories, while attention encodes neighboring states and observed social features.
- Final Position Clustering is an optional postprocessing method that reduces sampling bias and improves prediction quality when few samples are drawn.
- The paper reports state-of-the-art performance on the ETH/UCY and SDD benchmarks and multimodality capture in everyday and NBA scenarios.
- More than 10% improvement, and in some test cases more than 50%, is reported over existing trajectory prediction methods across ETH/UCY, SDD, and SportVU benchmarks.
2 Related Work
The paper focuses on human-human interaction modeling rather than scene-specific human-space prediction. Its approach combines stochastic recurrent modeling with backward posterior inference and observable-state social attention.
- Human-space models use scene environment information, whereas this work studies human-human trajectory prediction from dynamic interaction information.
- The timewise VAE backbone represents dynamic decision making that can lead to different trajectories at a given time.
- A backward RNN uses the whole ground-truth trajectory during learning to improve feature extraction for human-human interactions.
- Figure 1 depicts sequential stochastic latent variables, deterministic and stochastic states, social-feature attention, and training-only inference components.
- Neighbor attention relies on observable position and velocity, unlike approaches using hidden states that require consistently tracked neighbors.
3 Approach
SocialVAE predicts each agent’s future trajectory distribution from historical observations using a timewise VAE with recurrent stochastic dynamics, social attention, and backward posterior inference. Its training and sampling design models time-varying uncertainty while supporting variable numbers of neighboring agents and optional sample clustering.
- Problem formulation: SocialVAE estimates each agent’s future-position distribution from a joint historical observation of the scene.Prediction is performed independently for each agent using social features extracted from local observations, allowing scenes with arbitrary numbers of agents.
- Generative model: The model generates trajectories autoregressively as displacement sequences, which are accumulated from the observed position to obtain future spatial positions.Its generative formulation integrates over timewise latent variables and predicts each displacement conditioned on previous displacements, observations, and the latent state.
- Generative model: A timewise VAE introduces a latent variable at every prediction step and conditions the recurrent generative dynamics on the evolving RNN state.The model uses a conditional prior over each timewise latent variable rather than a fixed standard-normal prior, enabling flexible distribution tracking.
- Inference and training: A backward recurrent inference network uses the whole ground-truth trajectory during training to approximate posterior latent distributions at each future time step.At test time, the inference network is discarded; prediction uses only the generative model and observation encoder, without future-trajectory information.
- Inference and training: Training maximizes a timewise ELBO, while the final objective replaces displacement log-likelihood with accumulated-position squared error plus a KL divergence.Reparameterization enables optimization through sampled latent variables, and the position-based error accounts for accumulated trajectory deviations.
- Social encoding and sampling: Social attention encodes the target and neighboring agents from local observations, while Final Position Clustering selects diverse samples when only K predictions are retained.Attention is computed over the observation sequence using neighbors’ observed states; FPC oversamples, clusters final positions with K-means, and retains one representative trajectory per cluster.
4 Experiments
Experiments evaluate SocialVAE quantitatively and qualitatively across pedestrian and NBA movement benchmarks, then analyze its components through ablations. Results show competitive prediction errors, improved distribution quality, and benefits from FPC, attention, and timewise latent components.
- Quantitative Evaluation: SocialVAE+FPC improves around 9% on ADE and 13% on FDE over AgentFormer, and 5% on FDE over MemoNet.These comparisons apply to methods that do not require postprocessing; FPC also provides faster inference without extra memory storage.
- Quantitative Evaluation: SocialVAE predictive distributions achieve lower NLL than competing distributions on the reported evaluations, indicating higher probability on ground-truth trajectories.The NLL comparison is reported alongside ADE/FDE results for tested datasets.
- Qualitative Evaluation: FPC improves prediction diversity and removes samples from low-probability regions, while retaining partial coverage of difficult ground-truth turns.In the UCY Zara case study, heatmaps use 2,000 samples and compare SocialVAE with and without FPC.
- Qualitative Evaluation: Social attention shifts toward nearby or approaching neighbors and ignores faraway or stationary agents as interaction relevance changes over time.The attention maps compare the first and twentieth frames in the UCY students003 scene.
- Ablation Study: Combining timewise latents and backward posterior approximation yields average ADE/FDE improvements of 26%/11% in ablation studies, with attention further reducing error.Using all four components, including FPC, produces a considerable FDE decrease and SOTA performance in the reported comparisons.
5 Conclusion and Future Work
SocialVAE combines timewise stochastic prediction, social attention, and backward posterior approximation, with FPC improving limited-sample predictions. It performs strongly across pedestrian benchmarks and is also applicable to SportVU NBA data, while semantic scene information remains future work.
- Conclusion: SocialVAE uses attention-based social features, timewise VAE prediction with RNNs, and backward posterior approximation to extract navigation strategies.The backward RNN synthesizes whole trajectories for navigation feature extraction.
- Conclusion: Final Position Clustering improves prediction quality when only a limited number of trajectory samples are available.FPC clusters the final positions of predicted trajectories.
- Conclusion and Future Work: SocialVAE achieves state-of-the-art performance in most ETH/UCY and SDD test cases and is applicable to SportVU NBA data.The conclusion also identifies semantic scene information as a future avenue for more physically acceptable trajectories.
A Evaluation Metrics
The evaluation uses displacement-based errors and likelihood-based scoring to assess predicted trajectories and predictive distributions at ground-truth trajectories.
- Displacement Metrics: Average Displacement Error measures Euclidean distance between predicted and ground-truth positions averaged across all prediction frames.It averages over t = T + 1, ..., T + H.
- Displacement Metrics: Final Displacement Error measures Euclidean distance between the predicted position and ground-truth position in the final frame.The final frame is T + H.
- Likelihood Metric: Negative Log Likelihood scores the negative logarithm of the predictive PDF evaluated at ground-truth trajectories.The predictive distribution is estimated from 2,000 samples using Gaussian kernel density estimation, independently at each time step.
B Social Features
SocialVAE computes attention from three neighbor-related features: inter-agent distance, bearing alignment, and minimal predicted distance within a time horizon.
- Social Features: The first social feature is the Euclidean distance between an agent and its neighbor.This feature is computed for agent i and neighbor j at time t.
- Social Features: The second social feature is the cosine of the bearing angle from the focal agent to its neighbor.It captures the angular relationship used in attention computation.
- Social Features: The third social feature is the minimal predicted distance between agents within a default 7-second horizon.Together, these features support attention computation over neighboring agents.
C Data Acquisition of SportVU NBA Dataset
The SportVU NBA evaluation uses rebounding and scoring sub-datasets that capture dense cooperative, adversarial, and nonlinear player interactions. The data are split for training and testing and downsampled to accommodate faster basketball movement.
- Dataset Characteristics: The scoring dataset contains cooperative and adversarial player interactions, highly nonlinear motions, set plays, and offensive and defensive schemes.These interactions make the dataset a challenging trajectory-prediction setting.
- Preprocessing and Splits: Scenes are randomly split into testing and training sets using a 1:4 ratio, with 0.12-second frame intervals and 12-frame predictions from 8-frame observations.The basketball data are downsampled because players move faster than normal pedestrians.
D Additional Results on SDD
Table 6 reports ADE/FDE results on the Stanford Drone Dataset in meters, using the mean of the best 20 predictions.
- Table 6 evaluates performance on the Stanford Drone Dataset (SDD).
- The reported metrics are average displacement error and final displacement error.
- Results are measured in meters and summarized as the mean value across the best 20 predictions.
E Sensitivity Analysis on FPC
The FPC sensitivity analysis shows that increasing the sampling rate generally reduces ADE and FDE, with most gains appearing around a sampling rate of 20 before runtime costs grow.
- Figure 7 normalizes all values relative to sampling rate 1, which represents no FPC.
- ADE and FDE decrease roughly as the FPC sampling rate increases on ETH/UCY.
- About 10% ADE and 18% FDE improvement is achieved at a sampling rate around 20.
- Beyond sampling rate 20, further increases yield only about 2% extra improvement while increasing running time.
F Latent Space Analysis
The latent-space analysis tests whether SocialVAE organizes observed trajectories by heading and speed. Its latent distributions distinguish observations with semantically different features.
- Figure 8 visualizes latent-variable distributions for observations varying in trajectory speed and turn direction.
- The analysis uses 15 eight-frame observations formed from five trajectory headings and three constant speeds.
- For each observation, 150 latent samples are drawn from the prior at the first prediction time step.
- The model clearly distinguishes observations with semantically different features in the latent space.