PaperlayerView exact revision on arXiv

Source-linked AI summary

MTR++: Multi-Agent Motion Prediction with Symmetric Scene Modeling and Guided Intention Querying

Shaoshuai Shi, Li Jiang, Dengxin Dai, Bernt Schiele

arXiv:2306.17770v2cs.CV

TL;DR

Motion prediction must handle diverse agent behaviors and complex scenes, while prior goal-based approaches depend on costly dense candidate sets. The paper introduces MTR with learnable intention queries and MTR++ with symmetric scene modeling and mutually guided queries for multi-agent prediction. MTR reaches state-of-the-art WOMD performance, and MTR++ improves multi-agent accuracy and efficiency, though quality scoring and rare-scenario diversity remain challenges.

  • Problem

    Motion prediction must anticipate multimodal traffic-participant behavior in complex road contexts, while dense goal-candidate methods incur performance and computation trade-offs.

  • Method

    MTR uses transformer encoder-decoder learnable intention queries with global intention localization and local movement refinement; MTR++ adds symmetric scene context modeling and mutually guided queries for multiple agents.

  • Results

    The initial MTR achieves a +8.48% mAP gain over previous ensemble-free approaches on WOMD, while MTR++ predicts more accurate multimodal trajectories for multiple agents with increasing efficiency advantages as agent count grows.

  • Takeaways & Limitations

    Learnable intention queries reduce reliance on dense goal candidates, and MTR++ extends multimodal prediction to interacting agents with scene-compliant trajectories.

  • Takeaways & Limitations

    Quality scoring remains unreliable, and rare scenarios can produce homogenized trajectories that miss agents’ genuine intentions.

Abstract

from arXiv · show

Motion prediction is crucial for autonomous driving systems to understand complex driving scenarios and make informed decisions. However, this task is challenging due to the diverse behaviors of traffic participants and complex environmental contexts. In this paper, we propose Motion TRansformer (MTR) frameworks to address these challenges. The initial MTR framework utilizes a transformer encoder-decoder structure with learnable intention queries, enabling efficient and accurate prediction of future trajectories. By customizing intention queries for distinct motion modalities, MTR improves multimodal motion prediction while reducing reliance on dense goal candidates. The framework comprises two essential processes: global intention localization, identifying the agent's intent to enhance overall efficiency, and local movement refinement, adaptively refining predicted trajectories for improved accuracy. Moreover, we introduce an advanced MTR++ framework, extending the capability of MTR to simultaneously predict multimodal motion for multiple agents. MTR++ incorporates symmetric context modeling and mutually-guided intention querying modules to facilitate future behavior interaction among multiple agents, resulting in scene-compliant future trajectories. Extensive experimental results demonstrate that the MTR framework achieves state-of-the-art performance on the highly-competitive motion prediction benchmarks, while the MTR++ framework surpasses its precursor, exhibiting enhanced performance and efficiency in predicting accurate multimodal future trajectories for multiple agents.

1 INTRODUCTION

MTR addresses multimodal motion prediction with learnable intention queries and complementary global localization and local refinement. MTR++ extends this framework to simultaneous multi-agent prediction using symmetric scene modeling and mutually guided intention queries.

  • Motivation: Motion prediction must anticipate diverse agent behaviors amid complex road and environmental contexts for autonomous driving.The task uses observed agent states and road maps to predict future actions and support vehicle decision-making.
  • Motivation: Dense goal-candidate methods trade uncertainty reduction for candidate-density dependence, with too few candidates hurting performance and too many increasing computation and memory costs.MTR is introduced to reduce this dependence while improving multimodal prediction.
  • MTR Framework: MTR uses learnable intention queries in a transformer encoder-decoder, assigning queries to motion modes and combining global intention localization with local movement refinement.The queries represent groups of trajectories directed toward the same region and support efficient prediction.
  • MTR++ Framework: MTR++ extends MTR from single-agent to simultaneous multimodal prediction for multiple agents.Its design targets joint future trajectories rather than independently predicting one focal agent at a time.
  • MTR++ Framework: Symmetric scene context modeling and mutually guided intention querying let MTR++ model agent interactions and produce more accurate, scene-compliant multi-agent trajectories.The comparison figure also reports stronger inference speed and memory efficiency as the number of agents increases.
  • Experimental Results: The initial MTR achieves a +8.48% mAP gain over previous ensemble-free approaches on the WOMD motion prediction benchmark.The authors report this as a state-of-the-art result for the initial framework.

2 RELATED WORK

Related work covers scene-context encoding, multimodal behavior modeling, and multi-agent prediction. MTR combines locally connected transformer encoding with intention queries and continuous multimodal distributions, while MTR++ uses shared symmetric context and mutually guided queries.

  • Scene Context Encoding: Prior scene-context methods rasterize inputs or construct global polyline graphs, whereas MTR uses a locally connected transformer graph to preserve locality and improve memory efficiency.The approach supports larger map encodings for long-term motion prediction.
  • Multimodal Future Behavior Modeling: Existing multimodal behavior models generate trajectory samples, parameterize Gaussian mixture distributions, or use other compact representations of future motion.These approaches differ in whether they approximate trajectories explicitly or model a distribution parametrically.
  • Multimodal Future Behavior Modeling: Goal-based prediction can incur efficiency costs because dense goal candidates are numerous, motivating alternatives such as direct trajectory regression and intention-query modeling.The related-work discussion frames candidate density as a central computational concern.
  • Multimodal Future Behavior Modeling: MTR combines intention queries with Gaussian Mixture Models to produce a continuous, compact multimodal future distribution and occurrence probabilities for specified trajectories.This differs from sparse trajectory sets that do not provide a continuous future distribution.
  • Simultaneous Motion Prediction: Per-agent context normalization creates computational inefficiency for multi-agent prediction, while shared ego-centered encoding can disadvantage off-center agents.These limitations motivate symmetric scene-context modeling in MTR++.
  • Simultaneous Motion Prediction: M2I models pairwise interaction through influencer-reactor roles and sequential prediction, whereas MTR++ uses mutually guided intention queries for interactions among more than two agents.MTR++ integrates behavioral interaction directly into multimodal decoding.

3 MTR FOR MULTIMODAL MOTION PREDICTION

MTR uses a transformer encoder-decoder with local scene-context encoding and learnable intention queries to predict multimodal future motion. Its decoder jointly performs global intention localization and local movement refinement, while auxiliary future prediction adds interaction context.

  • Transformer Encoder for Scene Context Modeling: MTR encodes agent histories and road maps as polyline tokens, then applies stacked transformer layers with local self-attention to model scene relationships.The encoder preserves locality and supports larger scene-context encoding with improved memory efficiency.
  • Dense Future Prediction for All Agents: Dense future prediction supplies additional future-context features for all agents, and experiments show that this auxiliary task improves multimodal motion prediction.The predicted future positions and velocities are encoded and provided to the motion decoder.
  • Motion Decoder with Intention Query: The motion decoder uses stacked transformer decoder layers and learnable intention queries to iteratively refine multimodal future trajectories.Each query aggregates encoded context for trajectories associated with a particular intention point or motion mode.
  • Motion Decoder with Intention Query: MTR jointly optimizes global intention localization and local movement refinement to first identify an agent’s broad intent and then refine its trajectory.The two tasks are integrated within the intention-query motion decoder.
  • Motion Decoder with Intention Query: Each intention query corresponds to a specific motion mode, allowing MTR to use a minimal query set instead of densely placed goal candidates.The learnable embeddings stabilize training and enable efficient intention localization; the paper uses 64 queries in its setting.
  • Multimodal Prediction with Gaussian Mixture Model: The framework models future positions with Gaussian mixtures and trains using negative log-likelihood together with an L1 regression loss.The positive Gaussian component is selected by the intention query closest to the ground-truth trajectory endpoint, with cross-entropy maximizing its probability.

4 MTR++: MULTI-AGENT MOTION PREDICTION

MTR++ enables simultaneous multimodal motion prediction for multiple agents by replacing focal-agent-specific encoding with shared symmetric scene modeling and mutually guided intention queries. Its local-coordinate representations and query interactions support efficient, scene-compliant joint predictions.

  • Motivation: MTR++ addresses the computational inefficiency of encoding scene context separately for each focal agent during multi-agent prediction.The preceding focal-agent-centric strategy encodes the scene separately for each agent.
  • Framework Overview: MTR++ enables simultaneous motion prediction for multiple agents through shared symmetric scene context modeling and mutually guided intention querying.These are the two improvements introduced over MTR.
  • Symmetric Scene Context Modeling: The shared context encoder symmetrically encodes the entire scene for each agent, allowing its features to be reused by a motion decoder for any agent.This design supports simultaneous prediction without separately encoding the scene for every focal agent.
  • Symmetric Scene Context Modeling: Polyline features are transformed into polyline-centric local coordinate systems defined from agent motion or map geometry, rather than depending on a global coordinate system.Agent local frames use latest position and moving direction; map frames use geometry center and tangent direction.
  • Symmetric Scene Context Modeling: Query-centric self-attention models each token’s relationships in its own local coordinate system and has comparable computational cost to standard self-attention.It integrates token features with relative position and direction information for the query token.
  • Mutually-Guided Intention Querying: MTR++ transforms intention points into a common global coordinate system, lets intention queries interact, and feeds the updated queries into each focal agent’s decoder.The resulting information propagation guides agents’ multimodal decoding toward more informed and realistic future trajectories.

5 EXPERIMENTS

Experiments show that MTR and MTR++ improve marginal and joint motion prediction while combining strong accuracy with efficient multi-agent inference. Ablations attribute these gains to intention-query design, iterative refinement, symmetric context modeling, mutual guidance, and local positional encoding, although quality-score estimation remains limited.

  • Main Results: MTR improves marginal prediction over prior state of the art, increasing mAP by +8.48% and reducing miss rate from 15.11% to 13.51%.
  • Main Results: MTR++ further improves MTR’s marginal results on all metrics, including a +2.00% mAP gain from jointly considering multiple agents’ future behaviors.
  • Main Results: For joint prediction, MTR raises mAP from 12.39% to 20.37% and reduces miss rate from 49.42% to 44.11%, while MTR++ adds +2.89% mAP and cuts miss rate by 2.68%.
  • Efficiency: MTR++ (light) reaches 38.96% mAP with 11.7M parameters and 67ms latency, outperforming comparable or slower baselines in the reported comparisons.It improves over SceneTransformer from 27.88% to 38.96%, runs faster than DenseTNT at 67ms versus 540ms, and runs faster than HDGT at 67ms versus 1320ms.
  • Ablation Study: Learnable intention queries outperform latent-anchor and dense-goal methods by +5.53% and +4.67% mAP, respectively, through mode-specific explicit intention points.
  • Ablation Study: Iterative trajectory refinement improves mAP by +1.6% and reduces miss rate by 1.48% by aggregating trajectory-specific context features across decoder layers.
  • Ablation Study: Symmetric context modeling reduces latency and memory costs while preserving comparable performance, and mutual intention querying adds a +2.49% mAP improvement.
  • Ablation Study: Removing within-agent or cross-agent query interactions lowers mAP by at least −2.13%, while removing both lowers it by −2.70%.

6 CONCLUSION

MTR combines learnable intention queries with global intention localization and local movement refinement for multimodal motion prediction. MTR++ extends this design to multiple agents through symmetric scene context modeling and mutually-guided intention querying.

  • MTR uses a transformer encoder-decoder with learnable intention queries for multimodal future-trajectory prediction.
  • Global intention localization identifies an agent’s intent, while local movement refinement adaptively refines predicted trajectories.
  • MTR++ adds symmetric scene context modeling and mutually-guided intention querying to predict scene-compliant multimodal motion for multiple agents.
  • Experiments on the large-scale WOMD dataset show state-of-the-art performance on marginal and joint motion prediction benchmarks.
Loading 2306.17770v2…