Source-linked AI summary
Multi-Head Attention for Multi-Modal Joint Vehicle Motion Forecasting
Jean Mercat, Thomas Gilles, Nicole El Zoghby, Guillaume Sandou, Dominique Beauvois, Guillermo Pita Gil
TL;DR
The paper addresses vehicle forecasting under interactive, uncertain, and multimodal traffic behavior. It uses LSTM encoding and forecasting with multi-head self-attention to jointly predict Gaussian-mixture position distributions for all vehicles. The method outperforms compared models on forecast likelihood and produces interpretable interaction patterns.
Problem
Vehicle forecasting must account for interactions among multiple vehicles while representing unavoidable uncertainty and diverse future-position modes.
Method
An LSTM encoder-decoder with two multi-head self-attention layers jointly forecasts multimodal position distributions for all vehicles using vehicle observations.
Results
The model improves forecast likelihood over compared models, with lower miss rate and attention heads that specialize in interpretable interaction patterns.
Takeaways & Limitations
The approach combines joint interaction-aware forecasting, uncertainty estimation, and diversified multimodal predictions without predefined maneuver modes.
Abstract
from arXiv · showhide
This paper presents a novel vehicle motion forecasting method based on multi-head attention. It produces joint forecasts for all vehicles on a road scene as sequences of multi-modal probability density functions of their positions. Its architecture uses multi-head attention to account for complete interactions between all vehicles, and long short-term memory layers for encoding and forecasting. It relies solely on vehicle position tracks, does not need maneuver definitions, and does not represent the scene with a spatial grid. This allows it to be more versatile than similar model while combining any forecasting capabilities, namely joint forecast with interactions, uncertainty estimation, and multi-modality. The resulting prediction likelihood outperforms state-of-the-art models on the same dataset.
I. INTRODUCTION
Vehicle motion forecasting must handle interactive traffic, unavoidable uncertainty, and diverse future choices without relying on restrictive scene or maneuver representations. The paper combines attention-based interactions with likelihood-trained multimodal predictions.
- Motion forecasting supports autonomous-driving tasks such as path planning and target selection, but human driver behavior remains imperfectly predictable.
- Forecasting must represent interactions among a variable number of vehicles despite uneven observation accuracy and partial occlusion.
- Future-position PDFs combine local maxima for discrete choices with dispersion representing continuous uncertainties from perception, estimation, and modeling.
- Predefined maneuvers may not capture the complex and numerous modes in trajectory data, while sampling-based models do not directly express a PDF.
- The paper contributes multi-head attention for specialized interactions, long-range attention with joint multimodal forecasts, and diversified predictions trained by maximizing likelihood.
III. INPUTS AND OUTPUTS
The model maps sequences of all vehicles’ past positions to joint sequences of Gaussian-mixture forecasts. Its formulation supports variable vehicle counts and forecast horizons while preserving permutation equivariance across vehicles.
- Inputs: Inputs are fixed-frequency histories of all vehicles’ (x, y) positions, centered on the ego vehicle at the current time.Each history contains a fixed number of observations, nhist.
- Outputs: For every vehicle, forecast step, and mixture component, the output specifies predicted position, spread, correlation, and mixture weight.The sextuplet is (x̂, ŷ, σx, σy, ρ, p).
- Outputs: Mixture weights form an nmix-component probability distribution, with their sum constrained to 1.The weights satisfy ∑m=1^nmix pm = 1.
- Model mapping: A neural network predθ maps the defined input sets to output sets and is permutation-equivariant along the vehicle axis for any vehicle count and forecast horizon.The model is defined for all nveh and npred values.
IV. MODEL ARCHITECTURE
The forecasting model combines an LSTM encoder-decoder with two multi-head self-attention layers. Attention models current interactions and maintains coherence among forecast position sequences.
- Architecture: The architecture uses LSTM networks for encoding and forecasting within an encoder-decoder structure.The model is divided into Encoder, Self-Attention, Predictor, and Decoder components.
- Self-attention: The first self-attention layer follows encoding to incorporate interactions among vehicles at the current time.This attention layer is one of two added to the LSTM architecture.
- Self-attention: The second self-attention layer follows forecast-time unrolling so the predicted position sequences remain coherent with one another.The two attention layers have similar architectures but different weights.
- Encoder: The encoder estimates each vehicle’s current state from its past observations using an intermediary vector with a chosen dimension of 128.The encoded state is intended to represent position, kinematic state, and interaction features; the input sequences are processed by a one-dimensional convolutional layer.
C. Self-attention
The self-attention layers model interactions among vehicles through query-key matching and value aggregation, then feed interaction-aware sequences into forecasting.
- C. Self-attention: Each attention head projects vehicle features into value, key, and query tensors to select relevant information from other vehicles.Queries match keys through scaled dot products and softmax normalization.
- C. Self-attention: The resulting square attention matrix contains vehicle-to-vehicle coefficients, with entry (i, j) representing vehicle i attending to vehicle j.Softmax makes matching coefficients close to 1 and nonmatching coefficients close to 0.
- C. Self-attention: Outputs from all heads are concatenated, linearly combined, and added to the input through a residual connection.This combines the distinct feature selections produced by the heads.
- C. Self-attention: A second LSTM repeats attention features across prediction times, while another attention layer preserves coherence among time-dependent vehicle interactions before decoding.The decoder produces Gaussian mixture coefficients for each forecast time step.
V. ARCHITECTURE DISCUSSION
The architecture uses key-query self-attention to model interactions among a variable, unordered set of vehicles while jointly forecasting every vehicle.
- V. ARCHITECTURE DISCUSSION: Key-query self-attention accounts for interactions while allowing LSTM modules to operate on fixed-size inputs.The model accepts a varying number of interacting vehicles without imposing an ordering.
- V. ARCHITECTURE DISCUSSION: The model simultaneously forecasts every vehicle in the scene with vehicle-to-vehicle interactions.
- V. ARCHITECTURE DISCUSSION: Global attention has O(nveh^2) computation because it uses an nveh × nveh attention matrix.In the tested use cases, nveh is below 30 and usually around 10, making attention affordable.
- V. ARCHITECTURE DISCUSSION: Attention heads specialize in different interaction patterns and produce interpretable interactions.
B. Maneuver free multimodal forecast
The method forecasts maneuver-free multimodal position distributions, evaluates likelihood and diversity, and addresses limitations of NLL-based evaluation and optimization.
- B. Maneuver free multimodal forecast: The model outputs a sequence of position mixture density functions rather than a mixture of complete trajectory density functions.Mixture-component centers are treated as local maxima and can be tracked as forecast trajectories by matching similar coefficients.
- B. Maneuver free multimodal forecast: Its fixed mixture components diversify solely through minimizing the time-averaged negative log-likelihood over the data distribution.There is no explicit link between mixture modes at consecutive forecast times.
- B. Maneuver free multimodal forecast: RMSE, FDE, and NLL evaluate forecasting performance, but only NLL directly accounts for multimodality and none measures accuracy-diversity trade-offs fully.Miss Rate is additionally considered for this trade-off.
- B. Maneuver free multimodal forecast: Miss Rate measures the fraction of cases where all forecasts miss the final position by more than 2m.Adding relevant components lowers miss rate and indicates the accuracy-diversity trade-off.
- B. Maneuver free multimodal forecast: Table I compares MNLL, RMSE, FDE, and MR against baselines evaluated on the same dataset and functions.CSP(M) results were recomputed with minor modifications for fairness.
- B. Maneuver free multimodal forecast: NLL can overfit part of the output and become unreliable, so predicted standard deviations are clipped at a 10cm minimum.
VII. APPLICATION
The application uses the NGSIM US-101 and I-80 datasets and their published preprocessing to support fair comparison with prior results.
- VII. APPLICATION: The model was implemented in PyTorch using the NGSIM US-101 and I-80 datasets and preprocessing from published accompanying code.That preprocessing defines training, validation, and test splits and simulates observations from a selected vehicle.
A. Performance indicators comparison
The comparison evaluates forecasting models under shared dataset and evaluation conditions, highlighting likelihood, miss rate, and RMSE differences across model designs.
- All compared models except GRIP were trained and evaluated on the same dataset with the same functions.
- CSP uses maneuver classification and convolutional social pooling to condition multimodal forecasts for the center vehicle.
- CSP with unimodal forecasts produces better RMSE results than the multimodal CSP(M) model.
- GRIP achieves the best RMSE but does not account for error estimation or multimodality.
- SAMMP uses six mixture components to match CSP(M) for a fair comparison.
- SAMMP improves forecast likelihood over compared models, while its best-matching six-trajectory RMSE values are 0.31, 0.71, 1.20, 1.80, and 2.55.Its most-probable trajectory does not improve on GRIP’s RMSE, making that comparison unclear.
B. Attention interpretation
The learned attention matrices reveal recurring interaction patterns among vehicles, including front-vehicle, rear-vehicle, and lane-related specialization.
- One attention head often specializes in the vehicle directly ahead, or attends to itself when no front vehicle exists.
- In one training, a head specialized in rear-vehicle attention instead of front-vehicle attention.
- Another head often matches the closest front vehicle in any lane, while other heads combine many vehicles less clearly.
- Attention patterns often distinguish front from rear vehicles and differentiate lanes.
C. Multi-modal forecasting
The model represents multimodal future-position distributions and visualizes how attention and probability density capture interacting, uncertain driving outcomes.
- An aggressive overtaking scene illustrates diversified lateral possibilities, including aborted or less aggressive overtakes and possible perception errors.
- The attention visualization encodes vehicle-to-vehicle coefficients with arrows, self-attention circles, widths, angles, and purple-to-yellow matrix colors.
- The ego vehicle’s forecast is shown as superposed position probability density functions in blue log-scale shades against the green ground-truth future path.
- Unmodified NLL training produces multimodal outputs matching the possible outcomes and leads to lower NLL values without changing the forecast distribution.
- The model can be extended with additional vehicle observations or specialized encoders, predictors, and decoders for interacting object classes.These extensions are enabled by the forecasting algorithm’s model-free design.
- The NGSIM US-101 and I-80 datasets contain overhead views of straight highway roads, so they do not test road-network understanding.The architecture can be extended with lane-centerline inputs for more complex road-scene reasoning.
- Optimal transport paths between successive Gaussian mixtures could define infinitely many trajectories, or finitely many paths passing through local maxima.
IX. CONCLUSIONS
The paper proposes joint multimodal forecasting for all vehicles with interpretable social attention and reports improved NLL results, while identifying extensions for more complex scenes.
- The proposed solution jointly forecasts multimodal probability functions for all vehicles in a road scene.
- The method generates interpretable social attention coefficients that can be extended to other road-scene observations.
- The approach outperforms state-of-the-art results on the NLL indicator, supporting forecasting capacity and uncertainty evaluation.
- Future work will include vehicle attention to lanes and evaluation on the Argoverse dataset in urban situations.