Source-linked AI summary
M2I: From Factored Marginal Trajectory Prediction to Interactive Prediction
Qiao Sun, Xin Huang, Junru Gu, Brian C. Williams, Hang Zhao
TL;DR
Existing trajectory predictors often model agents independently, which can produce overlapping or mutually inconsistent multi-agent futures. M2I factorizes interactive prediction through relation classification, marginal influencer prediction, and conditional reactor prediction, achieving state-of-the-art performance on the Waymo Open Motion Dataset interactive benchmark.
Problem
Existing methods mainly predict individual-agent trajectories and may produce overlapping or mutually inconsistent trajectories when agents interact.
Method
M2I predicts interaction relations, then combines marginal predictions for influencers with conditional predictions for reactors in a modular framework.
Results
M2I achieves state-of-the-art performance on the Waymo Open Motion Dataset interactive prediction benchmark, with large mAP improvements over prior models and challenge winners.
Takeaways & Limitations
Factorizing relations between interacting agents enables scene-compliant joint prediction while retaining modularity across context encoders and prediction heads.
Takeaways & Limitations
M2I’s performance depends heavily on the amount of interactive training data, with smaller gains for agent types lacking sufficient interactive scenarios.
Abstract
from arXiv · showhide
Predicting future motions of road participants is an important task for driving autonomously in urban scenes. Existing models excel at predicting marginal trajectories for single agents, yet it remains an open question to jointly predict scene compliant trajectories over multiple agents. The challenge is due to exponentially increasing prediction space as a function of the number of agents. In this work, we exploit the underlying relations between interacting agents and decouple the joint prediction problem into marginal prediction problems. Our proposed approach M2I first classifies interacting agents as pairs of influencers and reactors, and then leverages a marginal prediction model and a conditional prediction model to predict trajectories for the influencers and reactors, respectively. The predictions from interacting agents are combined and selected according to their joint likelihoods. Experiments show that our simple but effective approach achieves state-of-the-art performance on the Waymo Open Motion Dataset interactive prediction benchmark.
1. Introduction
Existing marginal predictors can generate individually plausible but jointly inconsistent trajectories for interacting agents. M2I addresses this by factoring joint prediction into relation-aware marginal and conditional predictions, achieving state-of-the-art performance on the Waymo interactive benchmark.
- Marginal predictors may produce overlapping or unrealistic multi-agent trajectories because they do not reason about future interactions.
- Joint prediction becomes computationally difficult because the prediction space grows exponentially with the number of agents.
- M2I predicts scene-compliant trajectories by identifying influencer-reactor pairs and using marginal prediction for influencers alongside conditional prediction for reactors.
- M2I achieves state-of-the-art performance on the Waymo Open Motion Dataset interactive prediction benchmark.
2. Related Work
Prior trajectory-prediction work models multimodality and high-level intentions effectively for individual agents, but interactive prediction requires reasoning about agent relations and future trajectory dependence. M2I builds on this literature by explicitly identifying influencer-reactor relations and conditioning one agent’s prediction on another agent’s predicted future.
- Trajectory predictors commonly model multimodality with mixture models or generative models, while intention-conditioned models first predict goals, lanes, or maneuvers.
- Interactive prediction research includes learned interaction mechanisms and methods that discourage collisions or predict interaction types before generating trajectories.
- M2I explicitly identifies influencer-reactor relations to decouple interactive prediction and generate scene-compliant trajectories.
- Unlike approaches conditioning on a known vehicle or robot future, M2I conditions one agent’s trajectory on another agent’s trajectory that must itself be predicted.
3. Approach
M2I decomposes multi-agent trajectory prediction into relation-guided marginal and conditional predictions, reducing joint-distribution complexity while producing scene-compliant samples.
- 3.1. Problem Formulation: M2I factorizes interacting-agent prediction into a marginal distribution for an influencer and a conditional distribution for a reactor.The approach assumes the influencer behaves independently and the reactor responds to the influencer; non-interacting agents use separate marginal distributions.
- 3.3. Relation Predictor: M2I predicts relations among agents, assigns influencer and reactor roles, and uses those roles to decouple the joint prediction problem.The relation predictor classifies PASS, YIELD, or NONE using pass-yield behavior and assigns roles accordingly.
- 3.3. Relation Predictor: The relation predictor encodes observed agents, nearby agents, and map coordinates, then outputs a probability distribution over interaction types.Its modular encoder-decoder design can use existing context encoders, and training uses cross-entropy loss.
- 3.4–3.5. Trajectory Predictors: M2I generates influencer trajectories with a marginal predictor and reactor trajectories with a conditional predictor conditioned on each influencer future trajectory.The conditional model receives an augmented scene context containing the influencer trajectory and produces multimodal samples with the shared prediction architecture.
- 3.6–3.7. Sample Selection and Inference: For each influencer sample, M2I generates reactor samples, scores all N^2 pairs by joint likelihood, and retains K representative candidates.If the predicted relation is NONE, both agents use marginal prediction before the same selection step.
4. Experiments
M2I is evaluated on WOMD interactive prediction using relation prediction, marginal and conditional trajectory prediction, and joint sample selection. It outperforms strong baselines and improves prediction quality, scene compliance, and generalization across predictor models.
- 4.4.1 Validation Set: M2I outperforms both Waymo baselines on every reported validation metric and beats SceneTransformer on vehicle mAP and pedestrian miss rate.It also improves all-agent mAP, indicating more accurate confidence distributions and fewer false-positive predictions.
- 4.4.2 Testing Set: M2I improves mAP over SceneTransformer for vehicles and all agents on the interactive test set, while also outperforming past challenge winners and existing state-of-the-art models.The test comparison uses official benchmark results and reports a large-margin mAP improvement.
- 4.5.1 Relation Prediction: The relation predictor reaches 90.09% accuracy, and using predicted rather than ground-truth relations reduces vehicle mAP at 8s by 3.05%.This measures both relation classification quality and its downstream effect on trajectory prediction.
- 4.5.2 Conditional Prediction: Ground-truth influencer trajectories make conditional reactor prediction outperform the marginal predictor across all metrics, while predicted influencer errors reduce this advantage at inference time.M2I recovers overall performance by combining multiple influencer samples and selecting the most likely joint samples.
- 4.5.3 Generalizing to Other Predictors: M2I consistently improves all metrics, especially overlap rate and mAP, when its context encoder and prediction head are replaced with a different predictor model.The result supports generalization beyond the specific predictor architecture used in the main model.
- 4.6. Qualitative Results: In a challenging yielding and U-turn scenario, M2I identifies the interaction relation and predicts non-overlapping influencer and reactor trajectories, unlike the marginal baseline.The qualitative comparison links interaction modeling to improved accuracy and scene compliance.
5. Conclusion
M2I provides a modular joint-prediction framework using marginal and conditional predictors to exploit factorized relations between interacting agents. Its Waymo interactive benchmark results are state of the art, while limitations remain in minFDE, interactive-data dependence, and mutual-influence modeling.
- 5. Conclusion: M2I achieves state-of-the-art performance on the interactive Waymo Open Motion Dataset benchmark.The framework uses marginal and conditional predictors and exploits factorized relations between interacting agents.
- 5. Conclusion: The modular framework can generalize to a different predictor model, supporting use with varied prediction architectures.The authors report this generalization in the ablation study.
- 5. Conclusion: M2I still has a gap relative to state of the art on minFDE and depends heavily on interactive training data, especially for relation and conditional predictors.Vehicle improvements are larger than those for pedestrians and cyclists because interactive training scenarios are less available for the latter types.
- 5. Conclusion: M2I assumes no mutual influence between interacting agents, leaving more complicated mutually influential scenarios for future work.The authors observed an obvious influencer in almost all Waymo interactive scenarios but defer more complicated cases.
A. Additional Experiment Details
This section adds details on filtering interactive training data, training the baseline joint predictor, and training by agent types.
- A. Additional Experiment Details: The additional experiment details cover interactive-data filtering, baseline joint-predictor training, and training procedures by agent type.
A.1 Filtering Interactive Training Data
Interactive training scenarios are filtered by identifying pairs of interacting agents, while the baseline joint predictor enumerates selected goal pairs before trajectory completion.
- A.1 Filtering Interactive Training Data: The training filter identifies scenarios containing two interacting agents using the objects-of-interest mask.The filtering script is provided with the source code.
- A.1 Filtering Interactive Training Data: The baseline selects the top 80 goals for each agent and combines them into 6400 goal pairs for joint prediction.Each goal pair is processed by a two-layer MLP before subsequent trajectory modeling.
- A.1 Filtering Interactive Training Data: The trajectory completion model regresses full trajectories for both interacting agents from the predicted goal pairs.
A.3 Training by Agent Types
Training is separated by agent type because vehicles, pedestrians, and cyclists have different behavior models and imbalanced data distributions. Relation predictors are also grouped by interaction type, and additional examples compare M2I with a marginal baseline.
- A.3 Training by Agent Types: Marginal and conditional predictors are trained separately for vehicles, pedestrians, and cyclists because their behaviors and data frequencies differ.Vehicles account for 78% of the training data.
- A.3 Training by Agent Types: Relation prediction uses separate models for vehicle–vehicle, vehicle–pedestrian, vehicle–cyclist, and remaining agent-pair interactions.
- A.3 Training by Agent Types: Additional representative interaction examples illustrate M2I's advantage over the marginal baseline.
B.1 Influencer Overtakes Reactor
M2I correctly identifies overtaking relations and produces more accurate, scene-compliant trajectories than marginal prediction.
- M2I correctly predicts the overtaking relation and improves prediction accuracy and scene compliance over marginal prediction.
B.2 Reactor Yields to Influencer before Turning
M2I models yielding and merging interactions by predicting accurate reactor trajectories conditioned on the influencer, whereas marginal prediction overlooks these interactions.
- M2I correctly predicts yielding relations and accurate reactor trajectories, while marginal prediction produces less accurate trajectories by ignoring the interaction.
- M2I correctly predicts merging relations and reactor trajectories that follow the influencer, while marginal prediction can deviate far from ground truth.
C. Multi-Agent Generalization
M2I extends relation-based prediction to crowded multi-agent scenes, producing scene-compliant predictions across several interaction patterns.
- M2I provides scene-compliant relation predictions in crowded traffic involving more than two agents.
- For overtaking, M2I predicts the correct relation and avoids the overlapping trajectories produced by marginal prediction.
- For yielding before turning, M2I predicts the correct relation and accurate reactive reactor trajectories, unlike marginal prediction.
- For merging, M2I predicts reactor trajectories that follow the influencer, whereas marginal prediction can be far from ground truth.