PaperlayerView exact revision on arXiv

Source-linked AI summary

Motion Transformer with Global Intention Localization and Local Movement Refinement

Shaoshuai Shi, Li Jiang, Dengxin Dai, Bernt Schiele

arXiv:2209.13508v2cs.CV

TL;DR

Multimodal motion forecasting is challenged by direct regression’s shared representations and goal-based methods’ dependence on dense candidates. MTR uses learnable motion query pairs for global intention localization and local movement refinement, achieving state-of-the-art results on both WOMD challenges and ranking first on their leaderboards.

  • Problem

    Existing direct-regression methods converge slowly because motion modes derive from shared features, while goal-based methods incur efficiency costs that depend on candidate density.

  • Method

    MTR jointly performs global intention localization and local movement refinement with a small set of learnable motion query pairs, each assigned to a specific motion mode.

  • Results

    MTR achieves state-of-the-art performance on both marginal and joint WOMD motion-prediction challenges and ranks 1st on both leaderboards.

  • Takeaways & Limitations

    Mode-specific queries stabilize training without dense goal candidates, while iterative local refinement supports flexible and adaptive multimodal prediction.

  • Takeaways & Limitations

    MTR predicts one interested agent at a time, causing redundant context encoding for multiple interested agents in the same scene.

Abstract

from arXiv · show

Predicting multimodal future behavior of traffic participants is essential for robotic vehicles to make safe decisions. Existing works explore to directly predict future trajectories based on latent features or utilize dense goal candidates to identify agent's destinations, where the former strategy converges slowly since all motion modes are derived from the same feature while the latter strategy has efficiency issue since its performance highly relies on the density of goal candidates. In this paper, we propose Motion TRansformer (MTR) framework that models motion prediction as the joint optimization of global intention localization and local movement refinement. Instead of using goal candidates, MTR incorporates spatial intention priors by adopting a small set of learnable motion query pairs. Each motion query pair takes charge of trajectory prediction and refinement for a specific motion mode, which stabilizes the training process and facilitates better multimodal predictions. Experiments show that MTR achieves state-of-the-art performance on both the marginal and joint motion prediction challenges, ranking 1st on the leaderboards of Waymo Open Motion Dataset. The source code is available at https://github.com/sshaoshuai/MTR.

1 Introduction

Motion forecasting must capture multimodal agent behavior in complex scenes for safe autonomous driving. MTR addresses limitations of goal-based and direct-regression approaches with motion query pairs that jointly localize intentions and refine trajectories.

  • Motion forecasting jointly considers observed agent states and road maps despite inherently multimodal behaviors and complex scene environments.
  • Goal-based methods reduce trajectory uncertainty with dense candidates, but candidate density trades performance against computation and memory cost.
  • Direct-regression methods predict future trajectories without goal candidates, while MTR instead jointly optimizes global intention localization and local movement refinement.
  • MTR uses a small set of motion query pairs, assigning each pair to a specific motion mode for more stable and adaptive multimodal prediction.
  • MTR adds an auxiliary dense future prediction task to model future interactions and produce more scene-compliant trajectories for interacting agents.
  • MTR achieves state-of-the-art performance on WOMD marginal and joint benchmarks, with mAP gains of +8.48% and +7.98%, respectively.

2 Related Work

Prior motion-prediction work uses varied scene encoders and multimodal output strategies, including sampling, distributions, heatmaps, goals, and direct regression. MTR focuses on the underexplored motion decoder with a transformer-based design.

  • Earlier methods encoded maps and agent histories with rasterized images or structured representations such as lane graphs and vectorized scene elements.
  • Multimodal prediction strategies include trajectory sampling, Gaussian Mixture Models, heatmap sampling, action classification, and region-based training.
  • Goal-based methods estimate multiple destinations before completing a trajectory for each goal, whereas direct regression predicts trajectories from encoded agent features or latent anchors.
  • Goal-based strategies face efficiency concerns from many candidates, while direct regression can converge slowly when motion modes share agent features.
  • Recent high-performing WOMD methods mainly explore scene-context encoders, leaving motion-decoder design comparatively underexplored.
  • MTR introduces a transformer encoder-decoder with motion query pairs inspired by spatial anchor queries in DETR-style object detection.

3 Motion TRansformer (MTR)

MTR uses a transformer encoder-decoder framework that combines local scene-context encoding with motion-query-based global intention localization and iterative local trajectory refinement.

  • Transformer Encoder for Scene Context Modeling: MTR encodes agent histories and road-map polylines as tokens, then applies local self-attention to preserve scene locality and reduce memory use.The encoder outputs agent and map features for the motion decoder.
  • Transformer Decoder with Motion Query Pair: The decoder models multimodal prediction through motion query pairs, jointly optimizing global intention localization and local movement refinement.Static intention queries and dynamic searching queries serve the two complementary tasks.
  • Global Intention Localization: Static intention queries use spatially distributed intention points so each query predicts a specific motion mode without dense goal candidates.The intention points are obtained by applying k-means clustering to ground-truth trajectory endpoints; the setting uses 64 queries.
  • Local Movement Refinement: Dynamic searching queries are initialized from intention points and updated from predicted trajectories to gather trajectory-specific map features during refinement.The dynamic map collection module selects the L closest map polylines to each predicted trajectory.
  • Training Losses: MTR trains end-to-end with an auxiliary L1 regression loss and Gaussian negative log-likelihood losses applied across decoder layers.A hard-assignment strategy selects the closest motion query pair to the ground-truth endpoint as the positive Gaussian component.

4 Experiments

MTR is evaluated on WOMD’s marginal and joint motion prediction challenges, with ablations examining its decoder, dense future prediction, local attention, and query-pair training strategies. It achieves strong performance while showing benefits from mode-specific queries, iterative refinement, future interaction modeling, and local context encoding.

  • Experimental Setup: MTR is evaluated on WOMD’s marginal and joint motion prediction challenges, which predict six trajectories over eight seconds from one second of history.The marginal task evaluates agents independently, while the joint task evaluates two interacting agents’ future positions.
  • Main Results: MTR increases marginal-prediction mAP by +8.48% and decreases miss rate from 15.64% to 13.51% versus previous ensemble-free approaches.Its single-model result also achieves better mAP than MultiPath++, whose performance is boosted by model ensembling.
  • Main Results: MTR boosts joint-prediction mAP from 12.39% to 20.37% and decreases miss rate from 49.42% to 44.11%.The joint predictions are formed from combinations of two agents’ marginal predictions, retaining the top six joint predictions.
  • Ablation Study: Static intention queries with intention points improve mAP by +4.26% over latent learnable embeddings, while refinement raises mAP from 30.59% to 32.34%.The ablation attributes these gains to mode-specific querying and aggregation of fine-grained trajectory features.
  • Ablation Study: Dense future prediction improves mAP by +1.78%, supporting the value of modeling future interactions between agents’ trajectories.The auxiliary module supplies future context information to the decoder through encoded predicted future states.
  • Ablation Study: Local self-attention improves mAP by +0.77% with 256 polylines and +1.12% with 512, while global attention runs out of memory at larger context sizes.The reported comparisons use the same number of map polylines; local attention remains effective as map context increases to 1,024 polylines.

5 Conclusion

MTR frames multimodal motion prediction as joint global intention localization and local movement refinement, using learnable static queries and iterative refinement. It achieves state-of-the-art performance on both marginal and joint WOMD challenges, while remaining subject to agent-centric and post-processing limitations.

  • MTR jointly optimizes global intention localization and local movement refinement for multimodal motion prediction.Learnable static intention queries capture motion intentions, while iterative refinement probes fine-grained trajectory features.
  • MTR achieves state-of-the-art performance on both marginal and joint motion prediction challenges of the Waymo Open Motion Dataset.
  • The agent-centric strategy redundantly encodes context when predicting multimodal trajectories for multiple interested agents in one scene.The authors identify simultaneous multimodal prediction for multiple agents as future work.
  • Rule-based post-processing can produce suboptimal minADE and minFDE predictions when selecting a required number of trajectories from full multimodal outputs.The paper highlights selecting 6 trajectories from 64 full predictions as an open robustness issue.

A Implementation Details

MTR uses category-specific motion queries, encoded agent and map context, Gaussian regression losses, and clustered static intention points. Its inference procedure predicts multiple trajectories and applies endpoint-based NMS, with reported latency under batch-1 and batch-6 settings.

  • A single model predicts all three categories, with separate motion query pairs for Vehicles, Pedestrians, and Cyclists.
  • The input history state combines agent motion attributes, category masks, and time embeddings, while map features encode polyline geometry and types.
  • Gaussian regression losses from every decoder layer are summed with the auxiliary regression loss using equal weights.
  • 64 static intention points per category are generated by k-means clustering, with velocity and direction jointly represented in their spatial distribution.Vehicle intention points can share a heading direction while differing in velocity.
  • 59ms batch-1 latency and 33ms batch-6 latency are reported for WOMD scenes.The model predicts six trajectories for each of up to eight interested agents and one trajectory for up to 128 other agents.

B Per-class Results of MTR Framework

Per-category results are reported separately for marginal and joint motion prediction on WOMD validation and test sets, with the tables providing category-level reference values.

  • Marginal motion prediction: Per-category marginal motion prediction performance is reported on WOMD validation and test sets.
  • Joint motion prediction: Per-category joint motion prediction performance is reported on WOMD validation and test sets.

C Evaluation of Dense Future Prediction

The dense future prediction module predicts one 8-second trajectory for every validation-set agent, supporting future interaction modeling. Its evaluation reports ADE, FDE, and miss rate, while an additional table compares Argoverse2 leaderboard submissions.

  • 21.11% average miss rate at a 6.0m distance threshold is reported for dense future motion prediction.
  • The dense future prediction module is used to provide future interactions among agents for predicting multimodal trajectories of an interested agent.
  • The dense future prediction module predicts a single 8-second future trajectory for each agent in the WOMD validation set.ADE and FDE measure displacement error, while miss rate is evaluated from FDE at different distance thresholds.
  • Argoverse2 comparisons use the top-10 test-set submissions and vary K, the number of predicted trajectories used for evaluation metrics.

D Performance Comparison on Argoverse 2 Dataset

On Argoverse 2, MTR is evaluated using five seconds of history to predict six-second futures and achieves new state-of-the-art performance against leading submissions. The section also identifies local self-attention neighbor count as an encoder design factor.

  • MTR uses five seconds of scenario history to predict six-second future trajectories for one interested agent, with HDMap context available.
  • MTR achieves new state-of-the-art performance against the top-10 Argoverse 2 leaderboard submissions, with notable gains on miss-rate-related metrics.
  • Table 9 examines how the number of neighbors affects local self-attention in the transformer encoder.

E More Ablation Studies

The ablations study neighbor count, map-polyline count, decoder depth, and static-intention-point distribution. Results favor moderate local context and iterative refinement, while excessive map context can introduce noise.

  • Local self-attention neighbors: Increasing local-attention neighbors from 4 to 16 improves mAP, but 64 neighbors reduce mAP from 34.37% to 33.67%.The authors attribute the drop to weaker local-structure preservation and harder optimization with larger neighborhoods.
  • Dynamic map collection: Increasing collected map polylines from 32 to 128 consistently improves mAP, whereas collecting 256 causes a -0.41% mAP drop.The paper links the decline to additional noise and less accurate trajectory-specific map features.
  • Decoder layers: Increasing decoder layers consistently improves performance, and MTR uses 6 layers as a trade-off between performance and efficiency.Stacked decoder layers iteratively refine trajectories by aggregating increasingly accurate trajectory-specific features.
  • Static intention points: The static-intention-point ablation compares k-means generation of 64 points with uniform sampling of 8×8 = 64 points.

F Qualitative Results

Figure 7 provides additional qualitative results for the MTR framework.

  • Figure 7 presents more qualitative results of the MTR framework.

G Notations

This section provides a notation lookup and qualitative-figure conventions. The figure distinguishes interested agents with multimodal predictions from other agents with single predicted trajectories.

  • Table 13 provides a lookup table for the paper’s notations.
  • For each scene in Figure 7, MTR predicts 6 multimodal future trajectories for each interested agent, while other agents receive a single trajectory.
  • The notation i (o) denotes the predicted occurrence probability of the agent at spatial position o and time step i.
Loading 2209.13508v2…