Human Trajectory Prediction via Neural Social Physics

Jiangbei Yue; Dinesh Manocha; He Wang

Source-linked AI summary

Human Trajectory Prediction via Neural Social Physics

Jiangbei Yue, Dinesh Manocha, He Wang

arXiv:2207.10435v2Jul 21, 2022cs.CV

TL;DR

Trajectory prediction seeks methods that combine the interpretability of explicit behavioral models with the fitting flexibility of deep learning. The paper introduces Neural Social Physics, a neural differential-equation model with learnable physics and stochasticity modeling, and reports stronger benchmark performance, denser-scenario generalization, and plausible behavioral explanations. Its main limitation is that the physics model simplifies people as 2D particles.

Problem
Existing trajectory-prediction approaches trade off the explainability of explicit models against the data-fitting flexibility of deep learning.
Method
Neural Social Physics embeds a differentiable social-force-inspired physics model with learnable parameters in a deep network and models motion uncertainty with a Variational Autoencoder.
Results
NSP outperforms state-of-the-art methods across benchmark trajectory-prediction tasks, generalizes to higher-density unseen scenarios with plausible motions and fewer collisions, and provides plausible motion explanations.
Takeaways & Limitations
Embedding an explicit physics model gives trajectory prediction a supported combination of accuracy, generalization, and behavioral interpretability.
Takeaways & Limitations
NSP’s physics model overly simplifies people as 2D particles and does not capture factors such as affective states or dense-scenario interactions.

Abstract

from arXiv · show

Trajectory prediction has been widely pursued in many fields, and many model-based and model-free methods have been explored. The former include rule-based, geometric or optimization-based models, and the latter are mainly comprised of deep learning approaches. In this paper, we propose a new method combining both methodologies based on a new Neural Differential Equation model. Our new model (Neural Social Physics or NSP) is a deep neural network within which we use an explicit physics model with learnable parameters. The explicit physics model serves as a strong inductive bias in modeling pedestrian behaviors, while the rest of the network provides a strong data-fitting capability in terms of system parameter estimation and dynamics stochasticity modeling. We compare NSP with 15 recent deep learning methods on 6 datasets and improve the state-of-the-art performance by 5.56%-70%. Besides, we show that NSP has better generalizability in predicting plausible trajectories in drastically different scenarios where the density is 2-5 times as high as the testing data. Finally, we show that the physics model in NSP can provide plausible explanations for pedestrian behaviors, as opposed to black-box deep learning. Code is available: https://github.com/realcrane/Human-Trajectory-Prediction-via-Neural-Social-Physics.

1 Introduction

Trajectory prediction spans interpretable model-based methods and flexible deep learning, motivating Neural Social Physics (NSP), which combines an explicit learnable dynamics model with neural uncertainty modeling. NSP is reported to improve prediction accuracy, generalize to denser unseen scenarios, and provide plausible behavioral explanations.

Model-based methods offer explainability but weaker noisy-data fitting, whereas deep learning fits data effectively but can function as a black box.
NSP combines a differentiable social-force-inspired dynamics model with learnable parameters and a Variational Autoencoder for motion uncertainty.
NSP outperforms state-of-the-art methods across standard trajectory-prediction tasks, benchmark datasets, and metrics.
NSP generalizes to unseen higher-density scenarios while producing plausible motions with fewer collisions than pure black-box deep learning approaches.
The explicit model supports plausible explanations for pedestrian motions in addition to prediction accuracy and generalization.

2 Related Work

Trajectory analysis and prediction research includes statistical, simulation-based, and differentiable-equation approaches. Deep learning architectures increasingly model temporal, spatial, and interaction structure, while neural differential-equation research integrates learning with explicit dynamical systems.

2.1 Trajectory Analysis and Prediction: Trajectory analysis methods learn individual motion dynamics, latent patterns, and anomalies, but statistical models have limited capacity for large datasets.
2.1 Trajectory Analysis and Prediction: Deep trajectory-prediction research uses RNNs, GANs, CVAEs, CNNs, and GNNs to model temporal data, spatial features, and pedestrian interactions.
2.2 Crowd Simulation: Crowd simulation generates agent trajectories from initial positions and destinations using empirical models or data-driven methods.
2.3 Neural Differential Equations: Differentiable-equation research spans deep-learning-assisted solvers, differentiable physics, neural differential equations, and physics-informed neural networks.

3 Methodology

NSP combines a differentiable physics-inspired dynamical system with neural networks that estimate interpretable force parameters and trajectory stochasticity. NSP-SFM updates pedestrian states through goal attraction, inter-agent repulsion, and environmental repulsion, then adds learned noise for final predictions.

3.2 NSP-SFM: The neural differential equation combines learnable interpretable parameters with neural-network parameters and time-varying stochasticity conditioned on recent state history.The dynamics depend on the current state, neighborhood, environment, goal, and learned noise.
3.2 NSP-SFM: NSP-SFM models pedestrian acceleration as the sum of goal attraction, inter-agent repulsion, and environment repulsion forces.The formulation treats each pedestrian as a particle governed by Newton’s second law.
3.2 NSP-SFM: At each step, Goal-Network, Collision-Network, and the environment-force computation estimate forces before the model updates position and velocity.The resulting state is combined with noise estimated by the stochasticity module for the final prediction.
3.2 NSP-SFM: Goal attraction uses a destination-derived desired velocity, with a neural network estimating the adaptation time τ.The desired speed is updated as the pedestrian approaches the destination.
3.2 NSP-SFM: Collision interactions use a neural network to learn time-varying repulsion parameters, while environmental repulsion is learned from obstacles within a velocity-aligned view field.The environment parameter kenv is shared among obstacles, and the interpretable parameters include τ, knj, and kenv.
3.2 NSP-SFM: Training minimizes trajectory error and a CVAE loss for the distribution of stochasticity, using progressive training and a semi-implicit update scheme for stability.The deterministic force components are trained before the stochasticity component.

4 Experiments

NSP-SFM is evaluated against established trajectory-prediction baselines, under standard and ultra-sampling, denser unseen scenarios, interpretability analyses, and component ablations. It improves prediction accuracy and collision plausibility while exposing force-based and stochastic explanations for motion.

4.2 Trajectory Prediction: 16.94% and 10.46% improvements over Y-Net in ADE and FDE on SDD establish NSP-SFM’s standard-sampling benchmark advantage.On ETH/UCY, average improvements are 5.56% in ADE and 11.11% in FDE; Table 1 reports the standard-sampling comparison.
4.2 Trajectory Prediction: 70% and 62.5% average improvements over S-CSR in ADE and FDE on ETH/UCY show that NSP-SFM also leads under ultra-sampling.The comparison uses 20 samples per step and reports the overall minimum error.
4.3 Generalization to Unseen Scenarios: NSP-SFM generates fewer collisions than baselines on testing data and remains more plausible in unseen scenarios with substantially higher density.In the high-density experiments, baseline collision rates rise much more, while NSP-SFM’s rate can also increase but remains far more plausible.
4.4 Interpretability of Prediction: Force visualizations explain steering through goal attraction, inter-agent repulsion, and environmental repulsion, while the stochastic model captures unexplained trajectory deviations.The environmental force is prominent near a car, and randomness is illustrated when no pedestrian or obstacle accounts for the deviation.
4.4 Interpretability of Prediction: The force-based explanations are plausible rather than unique, so NSP-SFM is more explainable than black-box learning without claiming exclusive behavioral interpretations.The paper explicitly acknowledges that alternative explanations may exist.
4.5 Ablation Study: The ablation study shows that goal attraction alone performs well, additional repulsive forces help irregular geometries, and stochasticity substantially improves the full model.The three settings are Fgoal(w/o), NSP-SFM(w/o), and NSP-SFM(w), with the full model learning dynamics stochasticity through a per-step distribution.

5 Conclusions, Limitations, and Future Work

NSP is presented as a neural differential equation model that improves trajectory prediction, generalizes to denser scenarios, and provides plausible motion explanations. Its main limitation is that the physics model simplifies people into 2D particles.

NSP improves trajectory prediction, generalizes to significantly different scenarios, and provides possible explanations for motions.
Limitations: The main limitation is that NSP’s physics model overly simplifies people into 2D particles.
Future Work: Future work includes incorporating affective states, dense-scenario interactions, continuum models, reciprocal velocity obstacles, and learning-based collision detection.

A.1 Generalization to Unseen Scenarios

The generalization experiment evaluates trajectory plausibility in increasingly dense Coupa0 scenes using collision rate. NSP remains the best-performing method across tested agent counts and produces more plausible collision-avoiding predictions than the baselines.

Collision rate counts a collision when any agent pair reaches a distance below 2r during the evaluation period.The maximum possible number of collisions is N(N −1)/2.
NSP is always best in collision rate across 50, 74, 100, 150, and 200 agents, despite collision rates increasing with density.The comparison includes Y-net and S-CSR on Coupa0.
Y-net performs worse than S-CSR and NSP, while S-CSR’s number of collisions increases faster than NSP’s as agent count rises.

A.2 Interpretability of Prediction

NSP’s interpretability analysis visualizes how goal-directed, collision-avoidance, and environmental forces shape predicted trajectories. The examples connect deviations from direct goal motion to specific force contributions.

Figure 9 analyzes Fgoal, Fcol, and Fenv across complete trajectories and at two consecutive time steps.The figure uses separate colors for the three forces and marks other pedestrians at selected time steps.
Agents turn away from direct goal paths because incoming pedestrians produce strong collision-repulsion forces.
Environmental forces can redirect an agent even without nearby pedestrians, such as when grass acts as a weakly repulsive area.

A.3 Ablation Experiments

The ablations show that temporal modeling, the original MLP architectures, and progressively added model components contribute to NSP’s performance. The full design also models stochasticity through a per-step distribution.

Temporal modeling and the original MLPs produce the best ablation performance on SDD.
The interpretability visualizations include force contributions and predicted motion randomness.
Performance improves as components are incrementally added to the social force model baseline.The progression includes learning τ and k_nj and then the full model without CVAE.
The per-step distribution enables NSP-SFM to learn dynamics stochasticity and significantly improves results.The comparison includes Fgoal(w/o), NSP-SFM(w/o), and NSP-SFM(w).

B.1 Goal Sampling Network

The Goal Sampling Network combines scene segmentation and past-trajectory heatmaps to generate a spatial probability distribution over destinations.

B.1 Goal Sampling Network: Two U-nets form the Goal Sampling Network: Useg produces pixel-wise scene segmentation, and Ugoal predicts destination probabilities.The segmentation maps can be manually annotated or automatically generated, with GSN-derived maps providing an efficient option.
B.1 Goal Sampling Network: Trajectory heatmaps encode each observed position relative to scene-image coordinates before concatenation with the segmentation map.The heatmap construction converts the M past trajectories into M+1 spatial channels for Ugoal.
B.1 Goal Sampling Network: The network concatenates scene segmentation with trajectory heatmaps and outputs a normalized probability map over possible goals.The input has dimension H∗W∗(Kc+M+1), while each output pixel receives a probability between 0 and 1 and all probabilities sum to 1.

B.2 Conditional Variational Autoencoder

The CVAE models agent-specific dynamics stochasticity by learning residual corrections to deterministic physics-based predictions from motion history and latent variables.

B.2 Conditional Variational Autoencoder: The CVAE adds estimated stochasticity to a deterministic prediction based on goal, collision, and environmental forces.The deterministic prediction is computed with a semi-implicit scheme, after which the decoded residual produces the final prediction.
B.2 Conditional Variational Autoencoder: During training, the model encodes prediction error and recent trajectory history to parameterize a Gaussian latent variable.The error αt+1 is encoded as fbias, while the history (pt−7, ..., pt) is encoded as fpast before latent sampling.
B.2 Conditional Variational Autoencoder: At test time, latent samples from N(0, σlatentI) are decoded with motion history because the next ground-truth position is unavailable.The decoded stochasticity is added to the deterministic prediction in the same way as during training.

C Implementation Details

Implementation uses separate optimization settings for the force-related networks and CVAE, with dataset-specific hyper-parameters and environment-force formulations.

C Implementation Details: ADAM trains the Goal-Network, Collision-Network, and Fenv at 3 × 10−5–3 × 10−4, while CVAE training uses 3 × 10−6–3 × 10−5.CVAE training data is scaled by 0.005, and λ in lcvae is set to 1.
C Implementation Details: The model uses dataset-specific hyper-parameters summarized for all six datasets.The supplied implementation section explicitly points to Table 9 for the complete hyper-parameter settings.
C Implementation Details: Goal-Network parameter τ is constrained through a scaled sigmoid of a neural-network output, while scene segmentation uses dataset-specific class definitions.ETH/UCY uses walkable and unwalkable classes; SDD additionally includes weakly repulsive areas.
C Implementation Details: The environment force combines repulsion from ordinary obstacles with a weighted term for weakly repulsive obstacles.The formulation uses kenv and λweak to scale the respective obstacle contributions.