Source-linked AI summary
Multimodal Motion Prediction with Stacked Transformers
Yicheng Liu, Jinghuai Zhang, Liangji Fang, Qinhong Jiang, Bolei Zhou
TL;DR
Multimodal motion prediction must cover uncertain vehicle futures even though each scene supplies only one ground-truth trajectory. mmTransformer uses stacked transformers with independent proposals and region-based training, achieving first-place Argoverse leaderboard performance and competitive results while improving multimodal prediction.
Problem
A single ground-truth trajectory per scene makes it difficult to learn all plausible vehicle futures, while existing methods can rely on latent modes or proposal priors.
Method
mmTransformer uses stacked transformer modules to refine fixed independent trajectory proposals and region-based training to assign proposals to spatial modes.
Results
mmTransformer ranked 1st on the Argoverse leaderboard dated 16 Nov 2020 and remained competitive on the leaderboard.
Takeaways & Limitations
The framework improves multimodal prediction by combining contextual proposal features with training that encourages proposals to specialize in different modes.
Takeaways & Limitations
With vanilla training, increasing the number of proposals can cause modality collapse, while six proposals confine predictions to a local region around ground truth.
Abstract
from arXiv · showhide
Predicting multiple plausible future trajectories of the nearby vehicles is crucial for the safety of autonomous driving. Recent motion prediction approaches attempt to achieve such multimodal motion prediction by implicitly regularizing the feature or explicitly generating multiple candidate proposals. However, it remains challenging since the latent features may concentrate on the most frequent mode of the data while the proposal-based methods depend largely on the prior knowledge to generate and select the proposals. In this work, we propose a novel transformer framework for multimodal motion prediction, termed as mmTransformer. A novel network architecture based on stacked transformers is designed to model the multimodality at feature level with a set of fixed independent proposals. A region-based training strategy is then developed to induce the multimodality of the generated proposals. Experiments on Argoverse dataset show that the proposed model achieves the state-of-the-art performance on motion prediction, substantially improving the diversity and the accuracy of the predicted trajectories. Demo video and code are available at https://decisionforce.github.io/mmTransformer.
1. Introduction
Multimodal motion prediction is needed to represent uncertain vehicle behavior, but learning diverse outcomes from scenes with only one ground-truth trajectory remains difficult. mmTransformer addresses this with stacked transformers, independent trajectory proposals, and region-based training.
- Motivation: A single driving scene contains only one ground-truth trajectory despite multiple plausible future vehicle behaviors.This creates a challenge for learning to cover all possible outcomes.
- Motivation: Recent approaches use probabilistic or proposal-based formulations, with probabilistic methods relying on latent variables and predefined distributions.The supplied related-work passage introduces these two categories but is truncated before fully describing proposal-based limitations.
- Approach: mmTransformer uses stacked transformer encoder-decoder modules to hierarchically aggregate motion history, road information, and social interaction.The architecture refines randomly initialized trajectory proposals while incorporating contextual information.
- Approach: Independent decoder proposals aggregate contextual information through separate channels and make independent predictions, promoting diverse trajectory modes.The proposals are designed to carry customized features rather than identical representations.
- Approach: Region-based training assigns proposal sets to spatial regions and optimizes only the set containing the ground-truth trajectory.This encourages individual proposals to specialize in particular modes while preserving features learned by other proposals.
- Results: mmTransformer ranked first on the Argoverse leaderboard dated 16 Nov 2020 and remained competitive afterward.The contribution passage also reports substantial improvements from the model architecture and region-based training strategy.
2. Related Work
Motion prediction methods commonly extract contextual features before generating trajectories, while mmTransformer uses a fully transformer-based architecture to model multimodality at both feature and proposal levels.
- Motion Prediction: Deep learning motion-prediction pipelines represent surrounding information through rasterization or vectorization before extracting features and generating trajectories.Trajectory generation may be direct or based on prior knowledge.
- Motion Prediction: Feature-based methods primarily focus on extracting useful environmental information, whereas proposal-based methods generate multiple candidate trajectories.The supplied feature-based discussion is truncated after describing scene representation challenges.
- mmTransformer: mmTransformer combines feature-level and proposal-level multimodality, while region-based training reduces proposal correlation and preserves trajectory diversity.Explicitly incorporating modality into proposals also gives the pipeline a more interpretable structure.
- Transformer Methods: Unlike approaches using transformers mainly as feature extractors, mmTransformer employs a fully transformer-based architecture for multimodal motion prediction.Its design addresses the prediction problem rather than using attention only for sequence or interaction modeling.
- mmTransformer: The model overview combines stacked-transformer contextual aggregation with proposal decoding into trajectories and confidence scores.A trajectory generator produces trajectories, while a selector produces corresponding confidence scores.
3. Multimodal Motion Prediction Framework
The framework constructs multiple proposal features from scene context with stacked transformers, then uses region-based training to preserve diverse trajectory modes. Each proposal produces a trajectory and confidence score, while losses are applied to proposals associated with the ground-truth endpoint region.
- Framework: The model learns a proposal-feature set from motion history and surrounding context, then maps proposal features to multiple future trajectories.Each trajectory has T future steps and the prediction set contains K trajectories.
- Stacked Transformers: Stacked transformer units hierarchically aggregate motion, map, and social information, with each unit refining proposals from the preceding unit.The social constructor models dependencies among vehicles after individual vehicle and map features are encoded.
- Stacked Transformers: Distinct decoder proposals represent different modes because permutation-invariant transformers require proposals to remain distinguishable.Learned positional encodings initialize the proposal inputs, whose decoder outputs serve as proposal features.
- Proposal Feature Decoder: The proposal feature decoder has separate trajectory-generation and trajectory-selection branches, producing K predicted trajectories and K confidence scores.The trajectory branch applies an MLP to each proposal feature, while the scoring branch uses another MLP for confidence prediction.
- Region-based Training Strategy: Direct regression can cause mode averaging, while vanilla minimum-error training limits predictions locally and increasing proposal count can trigger modality collapse.These limitations motivate region-based training rather than selecting only the proposal closest to the ground truth.
- Region-based Training Strategy: RTS partitions endpoint space into non-overlapping regions and computes losses for all proposals assigned to the ground-truth region, avoiding interference across regions.The full objective combines Huber regression, KL-divergence scoring, and auxiliary region-classification losses.
4. Experiments
Experiments on Argoverse evaluate mmTransformer across benchmark performance, component ablations, partition strategies, proposal counts, and qualitative multimodal behavior. Results show strong accuracy and diversity, with explicit trade-offs from retaining diverse proposals.
- Dataset: Argoverse contains 340k five-second trajectory sequences; the task uses two seconds of history and context to forecast three seconds ahead.The dataset includes 205,942 training, 39,472 validation, and 78,143 testing cases.
- Results: The models achieve the best performance across all reported metrics, while 36-proposal RTS slightly lowers minADE and minFDE versus six-proposal mmTransformer.Post-processing discards redundant proposals to preserve diversity, reducing selected proposals in the ground-truth region.
- Component ablation: MR decreases from 23.3% to 10.6% when all transformer modules are applied, with map aggregation alone improving MR to 14.4%.The modules hierarchically aggregate motion, map, and social contextual information.
- Component ablation: RTS reduces MR from 17.6% to 9.2%, whereas vanilla training with many proposals cannot achieve comparable performance.The authors attribute the improvement to using many proposals while preserving modality information.
- Spatial partitions: Manual spatial partitioning slightly outperforms constrained K-means because it assigns blurry samples to more appropriate regions.The comparison uses six regions and six regional proposals per region.
- Number of proposals: Performance is best when the number of regions and proposals per region both equal six; imbalanced ratios hurt performance when proposal counts are small.With many proposals, changing the ratio yields marginal or worse gains.
- Visualization: Regional proposals specialize in their assigned areas, producing trajectories that end in preassigned regions and show low diagonal MR.Figure 5 visualizes endpoint groups and the MR matrix for regional proposals.
5. Conclusion
The conclusion presents mmTransformer as a transformer-based model for accurate multimodal motion prediction and introduces partition training to improve multimodal outputs.
- 5. Conclusion: mmTransformer uses a transformer-based framework for accurate multimodal motion prediction.The conclusion identifies the model as the paper’s central development.
- 5. Conclusion: A novel partition training method is introduced to improve multimodal prediction.The conclusion refers to the method as a contribution to the proposed framework.
- 5. Conclusion: Experiments show competitive results on the Argoverse benchmark.The conclusion reports the benchmark-level outcome without specifying individual metrics.
A. Implementation Details
The implementation uses standard transformer optimization and initialization settings, with fixed hidden dimensions and a defined local-region preprocessing setup.
- Optimization: mmTransformer is optimized with AdamW using a 1 × 10^-3 learning rate, 1 × 10^-4 weight decay, and 0.1 gradient maximum norm.All parameters use Xavier initialization.
- Architecture settings: Transformer modules use 128 hidden units, with two encoder and two decoder layers except for the social-constructor decoder.The social-constructor decoder contains four layers.
- Preprocessing: The implementation covers a 65m × 65m local region centered on the target agent and aligned with its last-observation heading.Data augmentation flips trajectories horizontally and randomly masks the first ten time steps.
B. Detailed Architecture
Each transformer module encodes contextual information and decodes refined trajectory proposals, using positional encodings to distinguish proposal queries.
- Transformer module: Surrounding information enters an encoder to produce contextual memory for proposal decoding.Spatial positional encodings are added to queries and keys at each multi-head self-attention layer.
- Transformer module: The decoder receives randomly initialized proposals, proposal positional encodings, and encoder memory to produce refined trajectory proposals.This design turns proposal refinement into the decoder’s output process.
C. Classification Loss
The classification loss raises scores for trajectory proposals assigned to the ground-truth region by aggregating scores within corresponding regions before cross-entropy calculation.
- The loss encourages trajectory proposals assigned to the ground-truth region to have higher scores.
- Each classification-logit term sums the scores belonging to its corresponding region.
- Cross-entropy penalizes the region-selection predictions using the ground-truth region index and an indicator function.
D. The Procedure of Partition Algorithm
The partition procedure normalizes ground-truth trajectory endpoints, divides them into equally sized clusters, and constructs regions from cluster convex hulls.
- The procedure first extracts all normalized ground-truth trajectory endpoints.
- Constrained K-means divides the endpoint samples into M clusters equally.
- A convex hull algorithm finds each cluster’s vertices, which are gathered to form the regions.
E. Inference
During inference, non-maximum suppression filters duplicated trajectories by ranking confidence scores and greedily retaining sufficiently distinct predictions.
- Predicted trajectories are sorted by confidence scores in descending order before selection.
- NMS greedily selects trajectories and excludes those close to any already selected trajectory.
- The selection-and-filtering cycle repeats until sufficient predicted trajectories are collected.