Source-linked AI summary
The Garden of Forking Paths: Towards Multi-Future Trajectory Prediction
Junwei Liang, Lu Jiang, Kevin Murphy, Ting Yu, Alexander Hauptmann
TL;DR
The paper addresses the difficulty of predicting and evaluating multiple plausible pedestrian futures when identical histories can lead to different paths. It introduces the Forking Paths benchmark and Multiverse, a model combining multi-scale location representations with graph-based recurrent decoding; the authors report state-of-the-art performance on both the multi-future dataset and VIRAT/ActEV.
Problem
Existing trajectory datasets usually provide only one observed future for a history, making quantitative evaluation of multi-future prediction difficult.
Method
The paper introduces the human-annotated Forking Paths dataset and Multiverse, which predicts multimodal future locations using multi-scale decoders and graph attention.
Results
The authors report state-of-the-art performance on the Forking Paths multi-future benchmark and the real-world VIRAT/ActEV benchmark.
Takeaways & Limitations
Forking Paths provides a quantitative benchmark for multi-future trajectory prediction, while Multiverse supports evaluation across multi-future and single-future settings.
Takeaways & Limitations
Performance drops when models trained on reconstructed simulation data are evaluated on real data because synthetic trajectories and images differ from the real domain.
Abstract
from arXiv · showhide
This paper studies the problem of predicting the distribution over multiple possible future paths of people as they move through various visual scenes. We make two main contributions. The first contribution is a new dataset, created in a realistic 3D simulator, which is based on real world trajectory data, and then extrapolated by human annotators to achieve different latent goals. This provides the first benchmark for quantitative evaluation of the models to predict multi-future trajectories. The second contribution is a new model to generate multiple plausible future trajectories, which contains novel designs of using multi-scale location encodings and convolutional RNNs over graphs. We refer to our model as Multiverse. We show that our model achieves the best results on our dataset, as well as on the real-world VIRAT/ActEV dataset (which just contains one possible future).
2. Related Work
Trajectory prediction research spans single-future methods and approaches that model uncertainty to generate diverse possible paths.
- Single-future trajectory prediction: Single-future methods predict one best pedestrian or vehicle trajectory, often using scene or visual features.
- Multi-future trajectory prediction: Multi-future approaches model trajectory uncertainty using inverse reinforcement learning, social pooling, generative adversarial networks, or variational auto-encoders.
3. Methods
Multiverse predicts multimodal future location distributions from observed trajectories and video, combining multi-scale spatial representations, recurrent decoding, graph attention, and continuous offsets.
- Model overview: Multiverse computes a multimodal distribution over future locations from past agent locations and video frames.
- History Encoder: The encoder represents locations on a 2D grid at multiple scales and processes them with semantic scene maps through a convolutional RNN.
- Fine Location Decoder: A fine decoder regresses a real-valued offset for each grid cell, combining coarse cell selection with continuous coordinates.
- Coarse Location Decoder: The coarse decoder predicts a belief-state heatmap over grid cells, while graph attention updates neighboring cell features with input-dependent weights.
- Design distinction: Unlike methods based on predefined anchor trajectories, Multiverse is not limited by fixed anchors.
- Training: Training supervises both the coarse belief state and regression offsets using classification and smoothed L1 losses, with regularization.
- Inference: Diverse beam search selects multiple future grid-cell sequences using a diversity penalty, then adds predicted offsets to form final trajectories.
4. The Forking Paths Dataset
Forking Paths is a human-annotated CARLA dataset designed to evaluate multiple plausible futures under identical observed histories, using reconstructed real-world scenes and dense labels.
- Motivation: Existing trajectory datasets observe only one of many possible futures, limiting quantitative evaluation of multi-future prediction.
- Scene reconstruction: The dataset reconstructs static scenes and dynamic elements from real-world ETH/UCY and VIRAT/ActEV videos in the near-realistic CARLA simulator.
- Human generation of plausible futures: Controlled agents are assigned meaningful reachable destinations, and human annotators generate natural alternative trajectories toward them.
- Sequence design: Sequences last up to 15.2 seconds, including 10.4 seconds of future motion for evaluating longer-term forecasts.
- Annotations: Each rendered view includes agent locations and pixel-precise semantic segmentation labels for 13 scene classes.
5. Experimental results
The experiments evaluate Multiverse and baselines on multi-future Forking Paths prediction and single-future VIRAT/ActEV prediction, including cross-domain and ablation settings. Results support the model’s performance and design choices while revealing concrete failure modes and simulation-to-real limitations.
- Evaluation setup: The evaluation covers multi-future prediction on Forking Paths and single-future prediction on VIRAT/ActEV, using standard and multi-future trajectory metrics.Metrics include ADE, FDE, minADEK, minFDEK, and NLL.
- Evaluation setup: The Forking Paths comparison uses Linear, LSTM, Social LSTM, Next, and Social GAN baselines alongside Multiverse.Models are trained on real VIRAT/ActEV videos and tested on synthetic CARLA-rendered videos.
- Main results: Multiverse achieves the best-published result on VIRAT/ActEV and compares favorably with state-of-the-art models across Forking Paths settings.The paper also evaluates negative log-likelihood on Forking Paths after quantizing trajectory outputs into a common grid.
- Generalization: Simulation-to-real evaluation lowers performance for all models, but preserves method ranking between real-data and simulation-data training.The authors interpret this consistency as supporting the proposed dataset for comparing relative method performance.
- Generalization: Synthetic data contain about 60% of the real trajectory data and non-photorealistic images, disadvantaging methods that rely on RGB input.Multiverse uses trajectories and high-level semantic segmentation and suffers the least performance drop under this domain shift.
- Ablation experiments: Ablations show performance drops when removing the spatial graph or fine location decoder, while using only the fine grid scale causes a smaller degradation.The reported metrics are ADE/FDE for single-future prediction and minADE20/minFDE20 for multi-future prediction.
6. Conclusion
The paper introduces the Forking Paths dataset and Multiverse model for multi-future forecasting, establishing a quantitative benchmark and reporting state-of-the-art performance on two challenging benchmarks.
- Forking Paths provides a quantitative benchmark and evaluation methodology for multi-future trajectory prediction using human-created continuations under identical pasts.
- Multiverse predicts multiple future locations with multi-scale location decoders and graph attention.
- The method achieves state-of-the-art performance on a large-scale real video dataset and the proposed multi-future trajectory dataset.