Source-linked AI summary
Multiple Futures Prediction
Yichuan Charlie Tang, Ruslan Salakhutdinov
TL;DR
Motion prediction must handle uncertain multimodal futures and interactions among varying numbers of agents. MFP learns latent modes and jointly predicts trajectories with dynamic state encoding, achieving state-of-the-art results on several vehicle-trajectory datasets.
Problem
Motion prediction must represent multiple possible futures caused by agent goals and interactions while handling variable agent counts and avoiding mode-averaging.
Method
MFP is a sequential probabilistic latent-variable model using discrete learned modes, shared parallel RNNs, dynamic attentional state encoding, and factorization for joint multi-agent prediction.
Results
MFP achieves state-of-the-art results on several vehicle trajectory datasets, including simulated and real vehicle data.
Takeaways & Limitations
MFP supports multimodal interactive prediction, closed-form trajectory probabilities, and hypothetical inference for planning from an ego agent’s future trajectory.
Takeaways & Limitations
The model assumes fully observable agent states represented as ground-plane (x, y) coordinates and uses discrete latent variables for persistent behavior modes.
Abstract
from arXiv · showhide
Temporal prediction is critical for making intelligent and robust decisions in complex dynamic environments. Motion prediction needs to model the inherently uncertain future which often contains multiple potential outcomes, due to multi-agent interactions and the latent goals of others. Towards these goals, we introduce a probabilistic framework that efficiently learns latent variables to jointly model the multi-step future motions of agents in a scene. Our framework is data-driven and learns semantically meaningful latent variables to represent the multimodal future, without requiring explicit labels. Using a dynamic attention-based state encoder, we learn to encode the past as well as the future interactions among agents, efficiently scaling to any number of agents. Finally, our model can be used for planning via computing a conditional probability density over the trajectories of other agents given a hypothetical rollout of the 'self' agent. We demonstrate our algorithms by predicting vehicle trajectories of both simulated and real data, demonstrating the state-of-the-art results on several vehicle trajectory datasets.
1 Introduction
Motion prediction must represent multiple possible futures, agent interactions, variable scene sizes, contextual information, and trajectory uncertainty. MFP addresses these needs with a probabilistic latent-variable framework for jointly modeling multimodal interactive futures and computing trajectory probabilities in closed form.
- Accurate motion prediction supports robust decision making, including high-level maneuvers and low-level path-planning optimization in autonomous driving.
- Multiple futures arise from differing agent goals and interactions, such as vehicles turning, going straight, or yielding during a merge.The framework must also scale to arbitrary agent counts and incorporate map and road context.
- Existing methods often capture only subsets of these requirements, including determinism, missing future interactions, explicit mode labels, or fixed agent counts.
- MFP is a sequential probabilistic latent-variable model trained from multi-agent trajectory data using a variational lower bound on log-likelihood.
- MFP jointly models multimodal interactive futures for all agents, scales to arbitrary agent counts through factorization, and computes conditional or unconditional trajectory probabilities without Monte Carlo sampling.
- Discrete latent variables learn semantically meaningful modes without labels, while parallel recurrent networks and dynamic attentional encoding model agent interactions and scene context.
2 Related Work
Prior motion-prediction methods include physics-based models and data-driven recurrent architectures, but commonly omit long-horizon interaction, multimodality, or scalable joint prediction. MFP is presented as a framework combining these capabilities.
- Classical physics-based and kinematic methods perform well over short horizons but suffer at longer horizons from limited interaction and context modeling.
- Data-driven approaches use RNNs or encoder-decoder models, including LSTMs, GRUs, CVAEs, and learned CNN-based input features.
- Prior multimodal methods either explicitly label modes or avoid end-to-end data-log-likelihood training, while many joint predictors independently roll out future trajectories.
- MFP is presented as a unifying framework that combines the desired features identified in the comparison with recent methods.
3 Multiple Futures Prediction
MFP models multimodal, interacting futures for a variable number of agents using discrete latent modes, sequential factorization, and shared recurrent encoders. Its dynamic state encoding represents surrounding-agent context for next-step prediction, while variational training learns from trajectories without latent-mode annotations.
- Problem formulation: MFP represents the joint states and future trajectories of a finite but variable number of interacting agents, with contextual scene information included through I_t.The joint state is X_t, future states are Y_δ, and each agent has a trajectory Y^n.
- Latent multimodality: Discrete latent variables z_n model agent-specific intentions or behavior modes, allowing multimodal futures without explicit mode labels.Each agent receives one Multinoulli latent variable whose values can represent behaviors such as left, right, straight, aggressive, or conservative.
- Scalable joint prediction: MFP factorizes the joint future across time and agents to reduce the exponential O(K^N) mode space and improve scalability.Each agent’s future depends explicitly on its own latent mode and implicitly on others through re-encoded predicted states.
- Scalable joint prediction: Shared-weight encoding and decoding RNNs provide each agent’s viewpoint while enabling efficient joint prediction for arbitrary numbers of agents.The architecture uses parallel RNNs per agent and combines their inputs in minibatches.
- Learning: Training maximizes a variational lower bound using an EM-like procedure, integrating out discrete latents so MFP learns directly from trajectory data.Because the latent variables have small cardinality, exact posteriors can be computed; trajectory log-likelihoods can be evaluated in O(K) without sampling.
- State encodings: A point-of-view transformation and dynamic attention state encoder aggregate relative positions of surrounding agents into features for recurrent prediction.Radial basis functions match and route relevant agents to the feature encoder.
4 Experimental Results
MFP is evaluated on simulated and real vehicle-trajectory data, including multimodal prediction, hypothetical rollouts, and planning. The experiments show learned semantic modes, strong trajectory-prediction performance, and more robust model-based planning under parameter variation.
- 4.1 CARLA: MFP learns semantically meaningful latent modes without explicit labels in CARLA intersection scenarios.The learned modes correspond to straight motion, stopping, and right turns.
- 4.1 CARLA: MFP significantly outperforms various other methods on the larger CARLA benchmark using jointly evaluated vehicle trajectories.The benchmark contains over 60K training sequences from two CARLA towns and reports minMSD at K=12.
- 4.2 NGSIM: NGSIM experiments predict five seconds of future motion from three seconds of history and report likelihood and trajectory-error measures.Multimodal models use minRMSE over five samples, which may be overly optimistic because ground truth selects the best trajectory.
- 4.2 NGSIM: MFP improves on multimodal CS-LSTM while using fewer than its six fixed maneuver modes on NGSIM trajectory prediction.The best performance combines interactive rollouts with dynamic attention encoding.
- 4.3 Argoverse: Preliminary Argoverse experiments use a three-mode MFP and validation results from dataset versions 1.0 and 1.1.The authors did not tune MFP hyperparameters for this dataset and expect additional tuning could improve performance.
- 4.4 Planning and Decision Making: MFP-based planning is more robust to test-environment parameter variations than the compared model-free RL baselines.The planner optimizes ego actions using progressively trained joint predictions for all three vehicles.
5 Discussions
The paper concludes that MFP supports joint, multimodal prediction for arbitrary numbers of agents and achieves state-of-the-art vehicle-trajectory results. It identifies mixed discrete-continuous latents and pedestrian or bicycle datasets as future directions.
- 5 Discussions: MFP facilitates joint multi-step temporal prediction for an arbitrary number of agents while learning latent modes directly from data.The framework uses interactive future rollouts with agent-specific point-of-view encoding.
- 5 Discussions: MFP demonstrates state-of-the-art performance on several vehicle trajectory datasets.Future work includes mixed discrete and continuous latent variables and evaluation on pedestrian or bicycle trajectories.
6 MFP Implementation Details
MFP uses agent-specific recurrent encoders and decoders, dynamic attention, discrete latent modes, and a variational training procedure. Its implementation normalizes trajectories, processes scene context, and supports arbitrary-agent comparisons through factorized representations.
- Architecture: MFP uses shared-weight parallel recurrent networks, with each network representing one agent’s viewpoint and temporal history.Bidirectional GRUs encode and decode two-dimensional agent positions.
- Dynamic encoding: Dynamic encoding maps agent states to keys, matches them with learnable slots using an RBF, and aggregates features through soft attention.The resulting slot representation is transformed into a 32-dimensional feature encoding.
- Latent modes: Discrete latent variables are projected through a Softmax to represent distributions over K behavior modes.One latent variable per recurrent network can learn semantically meaningful modes without explicit mode labels.
- Normalization: Trajectory normalization subtracts the average future position and each agent’s current position before prediction.This places each agent’s current position at (0.0, 0.0) in its own recurrent representation.
- Training procedure: Training samples trajectory clusters, normalizes them, encodes the scene, assigns one-hot latent modes, computes the loss, and updates parameters with backpropagation and ADAM.The algorithm includes an E-step and an update to the approximating prior.
- Model comparison: The model comparison distinguishes variational optimization, interactive rollouts, and hypothetical conditioning on an ego future trajectory.These criteria describe whether future agent predictions interact and whether they change under hypothetical ego behavior.
7 Carla Experiments
CARLA experiments use simulated three-vehicle intersection trajectories to evaluate MFP’s interactive multimodal predictions. The results illustrate that interactions can make one latent mode produce different behaviors and that MFP-4 predictions are accurate on Town01 and Town02.
- Dataset and setup: CARLA data simulate three vehicles in a Town-05 intersection for five seconds at 20 Hz, with randomized initial positions and maneuver modes.The red vehicle has three possible modes, while the far-left blue vehicle can go straight or turn right.
- Interactions: The same latent mode can produce different trajectories when another vehicle’s behavior changes the interaction.An aggressive vehicle may follow different paths depending on whether another vehicle cuts in front of it.
- Interactions: Interactive joint rollouts let MFP model more variation with fewer latent modes.The paper attributes this effect to future interactions influencing the trajectory associated with a given mode.
- Qualitative results: MFP-4 predictions on Town01 and Town02 test sets are described as very accurate and generally containing no more than two modes.The predictions are overlaid on Lidar point clouds, and the paper identifies attention-based state encoding as critical in this setting.
- Qualitative results: Town01 and Town02 quantitative minMSD results improved significantly over the previous paper version after increasing minibatch size and improving data normalization.The passage attributes the improvement to changing minibatch size from 1 to 8 and using better data normalization.
8 NGSIM Experiments
NGSIM experiments evaluate MFP on large real-vehicle trajectory splits using three seconds of history to predict five seconds ahead. Ablation results show that combining interactions with dynamic encoding gives the best test performance, alongside qualitative evaluations and implementation details.
- Dataset and setup: NGSIM experiments use six US-101 and I-80 sequences split into 70% training, 10% validation, and 20% testing.The resulting sample counts are 5922867 for training, 859769 for validation, and 1505756 for testing.
- Dataset and setup: The prediction task uses three seconds of past history to predict the next five seconds, with subsampling to 200 ms per timestep.The data are subsampled by a factor of 2.
- Evaluation: Table 6 results average five random trials and report the standard error of the mean, using MFP parameters saved after 300K updates.This evaluation protocol applies to all reported MFP models in the table.
- Training: NGSIM training uses ADAM with density-dependent minibatches averaging 60 samples, an initial learning rate of 0.001, and scheduled reductions.The learning rate is reduced by a factor of 10 every 100K updates and lower-bounded at 0.00005.
- Ablative studies: Figure 14 ablations show that the best NGSIM test performance uses both interactions and dynamic encoding.The figure examines the effects of these two components on test performance.
- Qualitative experiments: Additional qualitative experiments visualize the three modes learned by MFP-3 on NGSIM trajectories.Red, purple, and green denote the learned modes; blue and brown show previous trajectories, while orange shows the future trajectory.