Source-linked AI summary
Convolutional Social Pooling for Vehicle Trajectory Prediction
Nachiket Deo, Mohan M. Trivedi
TL;DR
Autonomous vehicles need reliable forecasts of surrounding-vehicle motion because traffic context and driver behavior make future trajectories difficult to predict. The paper introduces an LSTM encoder-decoder with convolutional social pooling and maneuver-conditioned multimodal outputs, reporting improved state-of-the-art performance on two public trajectory datasets.
Problem
Autonomous vehicles need reliable forecasts of surrounding-vehicle motion because traffic context, driver variability, multimodality, and vehicle interaction make future trajectories difficult to predict.
Method
The model combines an LSTM encoder-decoder, convolutional social pooling, and a maneuver-based decoder producing multimodal future-motion distributions.
Results
The model outperforms reported state-of-the-art results on two large publicly available vehicle-trajectory datasets.
Takeaways & Limitations
Convolutional social pooling more robustly models and generalizes spatial configurations of interacting vehicles while the model represents multimodal future motion by maneuver class.
Takeaways & Limitations
The approach relies purely on vehicle tracks and does not use complementary visual or map-based cues.
Abstract
from arXiv · showhide
Forecasting the motion of surrounding vehicles is a critical ability for an autonomous vehicle deployed in complex traffic. Motion of all vehicles in a scene is governed by the traffic context, i.e., the motion and relative spatial configuration of neighboring vehicles. In this paper we propose an LSTM encoder-decoder model that uses convolutional social pooling as an improvement to social pooling layers for robustly learning interdependencies in vehicle motion. Additionally, our model outputs a multi-modal predictive distribution over future trajectories based on maneuver classes. We evaluate our model using the publicly available NGSIM US-101 and I-80 datasets. Our results show improvement over the state of the art in terms of RMS values of prediction error and negative log-likelihoods of true future trajectories under the model's predictive distribution. We also present a qualitative analysis of the model's predicted distributions for various traffic scenarios.
1. Introduction
Vehicle trajectory prediction is needed for autonomous vehicles to plan safely in complex traffic, but surrounding-vehicle motion is difficult to model because of latent driver intentions, nonlinear behavior, multimodality, and interaction. The paper proposes an LSTM encoder-decoder with convolutional social pooling and a maneuver-based decoder to address these challenges.
- Reliable predictions of surrounding vehicles support autonomous-vehicle decisions such as lane changes, overtaking, and slowing for merging vehicles.
- Latent driver goals, varying driving styles, nonlinear long-horizon trajectories, multimodal behavior, and vehicle interactions make future-motion prediction challenging.
- Maneuver classes provide structure for representing multimodal future motion, while freeway lanes and travel direction help model vehicle interactions.
- The proposed approach is an LSTM encoder-decoder model for vehicle-motion prediction in freeway traffic.
- Convolutional social pooling replaces a fully connected layer over neighboring vehicles’ LSTM states, and the decoder predicts probabilities for six maneuver classes.
2. Related Research
Related research includes maneuver-based, interaction-aware, and recurrent-network approaches to vehicle motion prediction. These methods use maneuver recognition, handcrafted or learned interaction modeling, and recurrent architectures for trajectory or occupancy prediction.
- Vehicle-motion prediction research has been categorized into physics-based, maneuver-based, and interaction-aware models.
- Maneuver-based methods classify semantically interpretable behaviors and use maneuver recognition with maneuver-specific trajectory predictors.
- Interaction-aware methods incorporate inter-vehicle effects either through handcrafted relative-configuration costs or by learning interactions from trajectory data.
- Data-driven interaction modeling avoids reliance on handcrafted cost functions but requires large datasets to generalize across varied traffic configurations.
- Prior recurrent-network approaches use RNNs and LSTMs for maneuver classification, trajectory prediction, or occupancy-grid prediction at future time intervals.
3. Problem Formulation
The paper formulates trajectory prediction as estimating future positions from the target vehicle’s history and neighboring histories in a freeway-centered coordinate frame. It models future motion probabilistically through maneuver-conditioned distributions defined over selected lateral and longitudinal behaviors.
- 3. Problem Formulation: The target is the conditional probability distribution of a vehicle’s future positions given its own and neighboring vehicles’ track histories.
- 3.1. Frame of reference: A stationary frame centers the origin on the predicted vehicle, with the y-axis along freeway motion and the x-axis perpendicular to it.
- 3.1. Frame of reference: The coordinate choice makes the model independent of track-acquisition method and road curvature, supporting onboard sensing and freeway use with lane estimation.
- 3.2. Inputs and outputs: Model inputs are track histories containing x and y coordinates for the predicted vehicle and nearby vehicles within ±90 feet longitudinally and within adjacent lanes.
- 3.2. Inputs and outputs: The model outputs a probability distribution over the predicted vehicle’s future coordinates.
- 3.3. Probabilistic motion prediction: The conditional future-position distribution is expanded over maneuver classes to produce multimodal predictions.
- 3.3. Probabilistic motion prediction: At each future time step, a bivariate Gaussian represents future locations through its means and variances.
- 3.4. Maneuver classes: The maneuver scheme combines three lateral classes—left change, right change, and lane keeping—with two longitudinal classes—normal driving and braking.Lane changing is defined over ±4s around crossover, while braking means average horizon speed below 0.8 times prediction-time speed.
4. Model
The proposed model combines shared-weight LSTM encoders, convolutional social pooling, and a maneuver-based decoder to model vehicle dynamics, spatial interactions, and multi-modal future motion. Convolutional pooling preserves local spatial structure while the decoder predicts distributions across maneuver classes.
- Model overview: The model uses an LSTM encoder, convolutional social pooling layers, and a maneuver-based LSTM decoder.The encoder learns vehicle dynamics, the pooling layers capture spatial interdependencies, and the decoder produces a multi-modal predictive distribution.
- LSTM encoder: Shared-weight LSTMs encode the motion states of the predicted vehicle and surrounding vehicles from recent track histories.Each vehicle's final LSTM state is intended to represent its motion state.
- Convolutional social pooling: Convolutional social pooling replaces fully connected processing of the social tensor to preserve spatial relationships among neighboring vehicles.Fully connected layers make spatially adjacent and distant cells equivalent, which can hurt generalization across configurations.
- Convolutional social pooling: Convolutional and max-pooling layers are expected to learn locally useful features and add local translational invariance over the spatial grid.The social tensor is built from vehicle locations on a 13 × 3 lane-based grid, with rows separated by approximately one car length.
- Maneuver-based decoder: The decoder predicts future-motion distributions for six maneuver classes together with their class probabilities.The maneuver formulation accounts for the multi-modal nature of driver behavior.
- Training and implementation: The model is trained end to end using Adam with a learning rate of 0.001.The encoder LSTM has a 64-dimensional state and the decoder has a 128-dimensional state.
- Results: Over a 5 second prediction horizon, the proposed models outperform the baselines in both RMSE and NLL.Convolutional social pooling lowers RMSE relative to fully connected social pooling, while the maneuver-based decoder lowers NLL relative to a uni-modal prediction.
5. Experimental Evaluation
Experiments on NGSIM freeway-traffic datasets evaluate prediction accuracy, distributional fit, robustness to shifted spatial patterns, and qualitative behavior. Convolutional social pooling improves over fully connected pooling, while maneuver-conditioned predictions better represent multimodal futures despite differing RMSE and NLL behavior.
- Experimental setup: Experiments use publicly available NGSIM US-101 and I-80 freeway-traffic datasets, with trajectories recorded at 10 Hz across mild, moderate, and congested conditions.Prediction is evaluated over a 5-second horizon using RMSE and negative log-likelihood.
- Evaluation metrics: RMSE can favor averaging multiple modes, so negative log-likelihood is additionally reported to compare unimodal and multimodal predictive distributions.Averaging lane-change alternatives can produce a physically poor trajectory, motivating the distributional metric.
- Benchmark results: S-LSTM, CS-LSTM, and CS-LSTM(M) outperform the listed baselines in RMSE and NLL, while neighboring-vehicle motion provides a useful prediction cue.The compared systems include constant velocity, C-VGMM + VIM, GAIL-GRU, vanilla LSTM, and social-pooling variants.
- Pooling comparison: CS-LSTM outperforms S-LSTM on both RMSE and NLL, suggesting convolutional pooling better models interdependencies in vehicle motion than fully connected pooling.The comparison isolates the social-pooling design while retaining the LSTM prediction framework.
- Multimodal prediction: CS-LSTM(M) has higher RMSE but significantly lower NLL than CS-LSTM, indicating better fit to true trajectories through a multimodal predictive distribution.The RMSE calculation uses the highest-probability maneuver trajectory, which can be affected by maneuver misclassification.
- Robustness analysis: When training and test social tensors have non-overlapping spatial configurations, fully connected pooling drops sharply, whereas convolutional pooling is more robust to missing spatial patterns.The experiment removes odd ego-lane grid locations from training and even locations from testing, then compares RMS prediction errors.
6. Conclusions and Future Work
The paper presents convolutional social pooling for modeling interdependencies among neighboring vehicles and reports state-of-the-art performance on two public trajectory datasets. It also analyzes predicted distributions qualitatively and identifies reliance on vehicle tracks alone as a limitation.
- Conclusions: Convolutional social pooling uses convolutional connections to model interdependencies among neighboring vehicles’ motion.The approach is intended to model varied spatial configurations of interacting agents more robustly and generalize better.
- Conclusions: The model outputs multi-modal future-motion distributions organized by maneuver classes and provides qualitative analyses of those predictions.The analyses examine how surrounding vehicles affect predictive distributions in different traffic scenarios.
- Conclusions: The model outperforms reported state of the art on two large publicly available vehicle-trajectory datasets.
- Future Work: The approach relies purely on vehicle tracks to infer maneuver classes and future trajectories.The authors suggest incorporating visual and map-based cues as future work to improve maneuver classification and motion prediction accuracy.