Source-linked AI summary
Multi-modal Motion Prediction with Transformer-based Neural Network for Autonomous Driving
Zhiyu Huang, Xiaoyu Mo, Chen Lv
TL;DR
Autonomous-driving motion prediction must represent interacting agents and produce multiple interpretable future trajectories. The paper uses graph-based Transformer attention for agent interactions and multi-modal attention over map waypoints, with each attention mode conditioning a trajectory. Experiments on Argoverse report competitive accuracy, interpretability, and a simple structure and training process.
Problem
Motion prediction must represent social interactions and behavioral multi-modality while producing multiple possible trajectories with interpretability.
Method
The model represents interacting agents as a graph, uses Transformer attention for agent relations, and independently conditions trajectories on multi-modal attention over map waypoints.
Results
Argoverse experiments report competitive prediction accuracy, better interpretability, and a simple model structure and training process.
Takeaways & Limitations
Multi-modal attention can learn different agent-map relationships and identify possible map goals that condition diverse predicted trajectories.
Abstract
from arXiv · showhide
Predicting the behaviors of other agents on the road is critical for autonomous driving to ensure safety and efficiency. However, the challenging part is how to represent the social interactions between agents and output different possible trajectories with interpretability. In this paper, we introduce a neural prediction framework based on the Transformer structure to model the relationship among the interacting agents and extract the attention of the target agent on the map waypoints. Specifically, we organize the interacting agents into a graph and utilize the multi-head attention Transformer encoder to extract the relations between them. To address the multi-modality of motion prediction, we propose a multi-modal attention Transformer encoder, which modifies the multi-head attention mechanism to multi-modal attention, and each predicted trajectory is conditioned on an independent attention mode. The proposed model is validated on the Argoverse motion forecasting dataset and shows state-of-the-art prediction accuracy while maintaining a small model size and a simple training process. We also demonstrate that the multi-modal attention module can automatically identify different modes of the target agent's attention on the map, which improves the interpretability of the model.
I. INTRODUCTION
Motion prediction must capture road structure, agent interactions, kinematics, and behavioral uncertainty while producing multiple interpretable trajectories. The paper proposes Transformer-based graph and map-attention components, including multi-modal attention for diverse agent-map relationships.
- Traffic-trajectory prediction supports safe, informed, and human-like autonomous-vehicle decisions in complex traffic.
- The task is difficult because behavior depends jointly on road structure, agent interactions, kinematics, uncertainty, and multi-modal intentions.
- The framework represents interacting agents as a graph and uses Transformer attention to model their relationships.
- Multi-modal attention outputs independent agent-map attention results, each conditioning a possible trajectory, while only the closest-to-ground-truth head is updated.
- The model uses two cross-attention Transformer layers and is designed for simple training and deployment with satisfactory accuracy.
- The proposed waypoint-based Transformer network targets multi-modal prediction with improved accuracy and interpretability, and is evaluated on a large-scale real-world driving dataset.
II. RELATED WORK
Prior work commonly encodes driving scenes with rasterized images or vectorized graphs and uses generative, trajectory-set, goal-based, or proposal-based methods for multi-modal prediction. The paper instead learns map attention modes end to end, avoiding manually selected goals or proposals while retaining diversity and interpretability.
- Rasterized bird’s-eye-view scenes encode agents and road structure but require larger networks, more computation, and more training data.
- VectorNet uses polylines in a global interaction graph, whereas the paper separates map and agent-interaction processing with Transformer-based layers.
- Generative approaches such as CVAE and GAN can be hard to train and infer because repeated sampling is needed to recover plausible future-behavior distributions.
- Goal- and proposal-based methods provide diverse, interpretable outputs, but their goals or proposals are manually selected and require laborious processing and careful design.
- The proposed multi-modal Transformer learns attention to different map segments end to end, producing diverse trajectories while simplifying training and retaining accuracy, interpretability, and flexibility.
A. Problem Formulation
The task predicts K possible future trajectories and their probabilities from historical agent states and environmental context. The framework encodes agents and map waypoints, models agent-agent and agent-map relationships, then decodes trajectory and score outputs.
- A. Problem Formulation: The model input comprises the target agent’s and surrounding agents’ historical dynamic states plus current environment information.
- A. Problem Formulation: The formal objective maps input X to predicted trajectories through a parameterized prediction function f with parameters θ.
- B. Prediction Framework: The framework first converts raw map and agent states into feature vectors using map and agent encoders.
- B. Prediction Framework: An agent-agent encoder captures interactions among agents, while an agent-map encoder models the target agent’s attention across map segments.
- B. Prediction Framework: Interaction, map-attention, and target-agent dynamic features are concatenated before the scored trajectory decoder generates possible trajectories.
- B. Prediction Framework: The framework overview includes separate encoders for agents and waypoints, relationship modeling stages, and trajectory and score decoders.
- B. Prediction Framework: The detailed component structures are provided for the map encoder, agent-agent encoder, and agent-map encoder.
1) Map and Agent Encoders:
Agent histories are encoded from dynamic states, while map context is represented by waypoint and lane features. Shared encoders transform these inputs into features that preserve temporal motion and lane-level map information.
- Each agent state contains position, velocity, and heading, and histories are represented as tensors with shape (Th, 5).
- A 1D convolutional layer followed by an LSTM extracts temporal motion features using a shared encoder for target and surrounding agents.
- The model considers up to ten surrounding agents within 30 meters of the target agent.
- The map is represented by waypoints with relative coordinates and direction, augmented by lane turning, intersection, and traffic-control features.
- Waypoint features are combined with max-pooled same-lane information and encoded lane features to form final waypoint vectors.
2) Agent-agent Encoder:
The agent-agent encoder represents interacting vehicles as a graph and uses Transformer multi-head attention to model their relationships. The target agent queries the surrounding agents’ features to extract interaction information.
- All agents connect to the target agent, including a self-loop, while edge attributes are ignored.
- The Transformer encoder applies multi-head attention to encode interactions between the target and surrounding agents.
- The attention operation uses query, key, and value vectors with learned linear projections and scaled dot-product attention.
- The target agent’s feature supplies the query, while all agents’ feature vectors supply the keys and values.
3) Agent-map Encoder:
The agent-map encoder models the target agent’s relationships with map elements using multi-modal attention. Its mode-wise features condition diverse trajectory and score predictions.
- The target agent attends to all map elements, and a Transformer layer models the resulting agent-map relationship.
- Multi-modal attention outputs separate head results instead of concatenating and projecting them into one low-dimensional representation.
- Each attention mode produces a distinct feature representing a different relationship between the target agent and map.
- Predicted trajectories and scores condition on historical state, agent interactions, and map attention, with scores normalized into a probability distribution.
C. Training Objectives
Training combines trajectory regression and trajectory scoring objectives. The closest prediction receives the regression update, encouraging diverse mode-specific outputs.
- The model trains end-to-end with differentiable modules and uses smooth L1 loss across predicted trajectory time steps.
- The variety loss compares ground truth with only the closest predicted trajectory, selected by endpoint L2 distance.
- Only the attention head corresponding to the closest trajectory is updated, encouraging heads to attend to different map regions.
- The scoring objective uses cross entropy between ground-truth and predicted trajectory distributions.
- The total training loss is a weighted sum of scoring and trajectory regression losses, balanced by α.
1) Dataset:
The model is evaluated on the Argoverse Motion Forecasting dataset using displacement, probability-quality, and miss-rate metrics. Experiments use lane-waypoint maps, nearby agents, and a compact six-mode configuration.
- Dataset: The Argoverse dataset contains 324,557 real-world driving scenarios with five-second object trajectories sampled at 10 Hz.
- Metrics: Evaluation reports minADE, minFDE, brier-minFDE, and miss rate, covering trajectory distance, probability quality, and endpoint failures.
- Qualitative results: Figure 3 uses red for target history, blue for surrounding agents, yellow for predictions, and green for ground truth.
- Configuration: The model searches 40 nearby lanes with 10 waypoints each and 10 neighboring agents within 30 meters.
- Configuration: The encoders use six attention heads and six multi-modal modes, producing environment features shaped as (6, 768).
- Training setup: Training uses 100 epochs, batch size 64, and an RTX 2080Ti, with one epoch taking about 10 minutes.
3) Training:
The model produces diverse, map-compliant trajectories while visualizing mode-specific attention on map waypoints. Its attention patterns align left-turn and straight predictions with corresponding lane goals.
- Qualitative trajectory results: The model generates multiple trajectories that remain diverse and compliant with map structure across complex urban scenarios.The best prediction is close to ground truth, while alternatives vary in speed profiles or directions.
- Attention visualization: Attention scores are visualized on map waypoints separately for each prediction mode, with darker red indicating greater attention.Waypoints with scores greater than 0.01 are displayed.
- Mode-specific map attention: Left-turn predictions assign more attention to left-turn lanes, while go-straight predictions attend more to straight lanes.
- Interpretability: The multi-modal attention mechanism automatically extracts possible map goals, conditioning predicted trajectories on different goals.
2) Quantitative Results:
On the Argoverse test benchmark, the proposed model achieves the best accuracy on two distance-based metrics while using a smaller model and simpler training process. Its brier-minFDE and miss rate remain slightly worse than HOME.
- Benchmark comparison: The proposed model achieves the best prediction accuracy on the distance-based metrics minADE and minFDE.
- Benchmark comparison: The model's brier-minFDE and miss rate are slightly worse than HOME's because its scoring-loss weight requires further tuning.
- Efficiency and practicality: The proposed approach combines competitive accuracy with smaller model size and simpler training, easing preprocessing, postprocessing, training, and inference.
- Overall outcome: The paper reports competitive accuracy, better interpretability, and a simple structure and training process from experiments on the Argoverse motion dataset.