Source-linked AI summary
Multi-Agent Tensor Fusion for Contextual Trajectory Prediction
Tianyang Zhao, Yifei Xu, Mathew Monfort, Wongun Choi, Chris Baker, Yibiao Zhao, Yizhou Wang, Ying Nian Wu
TL;DR
Trajectory prediction must capture variable-agent interactions, scene constraints, and stochastic human behavior for autonomous-vehicle planning. MATF spatially fuses encoded agent histories and scene context, then uses adversarial training for stochastic predictions. Experiments report state-of-the-art results for highway driving and comparable pedestrian performance.
Problem
Trajectory prediction must handle stochastic behavior, social interactions, scene context, and varying numbers and configurations of agents.
Method
MATF spatially aligns encoded agent histories and scene context, applies convolutional fusion to model spatial interactions, and uses adversarial training for stochastic predictions.
Results
MATF achieves state-of-the-art long-range vehicle prediction on NGSIM and comparable performance with published results on a state-of-the-art pedestrian dataset.
Takeaways & Limitations
The architecture bridges agent-centric and spatial-centric trajectory prediction while retaining spatial structure for multi-agent and scene reasoning.
Takeaways & Limitations
Performance factors and the advantages and disadvantages of different architectures remain to be examined.
Abstract
from arXiv · showhide
Accurate prediction of others' trajectories is essential for autonomous driving. Trajectory prediction is challenging because it requires reasoning about agents' past movements, social interactions among varying numbers and kinds of agents, constraints from the scene context, and the stochasticity of human behavior. Our approach models these interactions and constraints jointly within a novel Multi-Agent Tensor Fusion (MATF) network. Specifically, the model encodes multiple agents' past trajectories and the scene context into a Multi-Agent Tensor, then applies convolutional fusion to capture multiagent interactions while retaining the spatial structure of agents and the scene context. The model decodes recurrently to multiple agents' future trajectories, using adversarial loss to learn stochastic predictions. Experiments on both highway driving and pedestrian crowd datasets show that the model achieves state-of-the-art prediction accuracy.
1. Introduction
MATF addresses trajectory prediction’s variable-agent, socially interactive, scene-constrained, and stochastic nature by spatially fusing agent histories with scene context. It predicts all agents jointly and achieves state-of-the-art accuracy across highway and pedestrian datasets.
- Trajectory prediction must account for stochastic motion, goals, social interactions, scene context, and varying numbers and configurations of agents.
- MATF combines agent- and spatial-centric approaches in an end-to-end encoder-decoder network for social and scene context.
- The Multi-Agent Tensor spatially aligns each agent’s encoded history with encoded scene context, preserving relationships between agents and scene features.
- Fully convolutional fusion captures spatially local interactions among agents and between agents and the environment while retaining the layout of all agents.
- Conditional generative adversarial training represents multimodal uncertainty in future trajectories, including alternative maneuvers and execution profiles.
- Experiments on driving and pedestrian datasets report state-of-the-art accuracy for highway driving and pedestrian trajectory prediction.
2. Related Work
Related work spans hand-crafted interaction models and deep networks using agent-centric aggregation or spatial-centric representations. Recent methods also model stochastic futures through generative distributions or maneuver-conditioned trajectory sets.
- Traditional trajectory methods use hand-crafted interaction features or cost functions, whereas recent deep networks generally outperform them.
- Agent-centric approaches aggregate multiple agents’ feature vectors, but pooling can omit distant interactions or discard agent uniqueness.
- Spatial-centric approaches preserve agent and scene geometry, but prior methods may bottleneck spatial information, omit scene context, or predict one target agent.
- Because human behavior is stochastic, newer approaches predict trajectory distributions using VAEs, inverse optimal control, conditional GANs, or stochastic policies.
- Other methods generate multiple possible trajectories by conditioning on maneuver classes rather than modeling a single deterministic future.
3. Method
MATF independently encodes multiple agent histories and overhead scene context, aligns them spatially, and applies shared convolutional fusion to model interactions. Residual agent representations are recurrently decoded, with adversarial training enabling stochastic multi-agent predictions.
- MATF accepts multiple agents’ past trajectories and an overhead static scene, then outputs predicted future trajectories for all agents.
- Shared LSTM encoders process each agent history while a CNN encodes scene context, allowing invariance to the number of agents.
- Agent vectors are placed at agents’ last observed positions and concatenated with the spatially aligned scene feature map to form the Multi-Agent Tensor.
- Fully convolutional U-Net-like layers fuse the tensor across spatial scales, modeling agent-agent and agent-scene interactions while retaining spatial locality.
- Fused vectors are sliced for each agent, added residually to its original encoding, and decoded by shared LSTM decoders into future trajectories.
- Training combines reconstruction and adversarial losses, with λ weighting their relative contributions.
- The conditional generator uses all agents’ histories, scene context, and Gaussian noise to produce stochastic future trajectories, while the discriminator distinguishes real from generated futures.
4. Experiments
The experiments evaluate MATF on driving and pedestrian datasets using ablations, quantitative comparisons, and qualitative analysis. Baselines isolate scene, multi-agent, deterministic, and stochastic components.
- Experiments use NGSIM, Massachusetts driving, ETH-UCY, and Stanford Drone datasets for training and evaluation.
- The evaluation includes quantitative comparisons with state-of-the-art alternatives and qualitative results for further analysis.
- The datasets cover freeway traffic, pedestrian crowds, and university-campus scenes with multiple interacting agent types and static context.
- Ablative studies compare simple LSTM, single-agent scene, multi-agent, full deterministic, and stochastic GAN variants.
- The GAN baseline samples multiple trajectories and reports the best L2 trajectory, using N = 3 for Section 5.1 and N = 20 for Section 5.2.
5. Results
Across driving and pedestrian datasets, MATF improves trajectory prediction by combining scene and multi-agent information, with strong deterministic and stochastic performance. Ablations show that both scene context and multi-agent interactions contribute, while spatial resolution affects accuracy.
- Driving datasets: MATF GAN performs at the state-of-the-art level on NGSIM, with particularly improved performance at longer 3-5s prediction horizons.Social Conv uses auxiliary maneuver-label supervision, whereas MATF does not require it.
- Driving datasets: MATF Multi Agent Scene consistently outperforms Single Agent Scene and Multi Agent on the Massachusetts driving dataset.Both Single Agent Scene and Multi Agent also outperform the LSTM baseline.
- Ablations: The results support modeling both agent-agent and agent-scene interaction: scene fusion learns contextual constraints, while the multi-agent model learns multi-agent interaction.
- Pedestrian datasets: MATF performs best in both deterministic and stochastic settings on ETH-UCY datasets.
- Pedestrian datasets: MATF Multi Agent Scene outperforms other deterministic models in ADE on Stanford Drone, while MATF GAN performs close to the state-of-the-art level.Social LSTM achieves the best FDE among deterministic models, while Sophie performs best among stochastic models.
- Ablations: The ideal Multi-Agent Tensor spatial resolution is 32 × 32, with a U-shaped performance curve caused by underfitting at low resolution and overfitting at high resolution.
6. Discussion
MATF combines scene constraints and social interaction while preserving the spatial structure of multiple agents and the scene. Across driving and pedestrian tasks, it achieves strong trajectory-prediction performance, while future work targets interpretability and multimodal structure.
- MATF models scene context constraints and social interaction while retaining the spatial structure of multiple agents and the scene.The architecture bridges agent-centric and spatial-centric trajectory-prediction approaches.
- In NGSIM, MATF achieved state-of-the-art long-range vehicle trajectory prediction, while Massachusetts ablations found both scene and multiagent representations necessary in complex contexts.The Massachusetts scenes included greater lane curvature and more entrances and exits than NGSIM.
- MATF’s adversarially trained stochastic model performed best relative to the cited maneuver-based approach, supporting distributional representations over maneuvers.The paper contrasts implicit maneuver distributions in MATF with the explicit representation used by the cited approach.
- On the Stanford Drone pedestrian dataset, MATF achieved comparable performance with previously published results despite using a novel spatial-centric architecture.Some recent models reported greater accuracy but used dramatically different architectures.
- Future work proposes unsupervised structured maneuver representations to improve prediction interpretability and capture multimodal agent-scene and agent-agent interactions.The proposed extension is presented as a planned direction rather than an evaluated result.