MotionLM: Multi-Agent Motion Forecasting as Language Modeling

Ari Seff; Brian Cera; Dian Chen; Mason Ng; Aurick Zhou; Nigamaa Nayakanti; Khaled S. Refaat; Rami Al-Rfou; Benjamin Sapp

Source-linked AI summary

MotionLM: Multi-Agent Motion Forecasting as Language Modeling

Ari Seff, Brian Cera, Dian Chen, Mason Ng, Aurick Zhou, Nigamaa Nayakanti, Khaled S. Refaat, Rami Al-Rfou, Benjamin Sapp

arXiv:2309.16534v1Sep 28, 2023cs.CVcs.AIcs.LGcs.RO

TL;DR

Reliable multi-agent motion forecasting must represent interactions rather than independent agent marginals. MotionLM discretizes trajectories into motion tokens and autoregressively decodes joint futures with temporal causality, achieving state-of-the-art WOMD interactive-prediction performance. Its conditional forecasts are designed to better reflect causal interventions, although acausal conditioning can yield higher scores from nonsensical reactions.

Problem
Independent per-agent marginals omit future dependencies between agents, producing inconsistent scene-level forecasts that are insufficient for planning.
Method
MotionLM casts multi-agent forecasting as language modeling over discrete motion tokens, using a single temporally causal autoregressive decoder to sample joint futures without anchors or latent-variable optimization.
Results
6% improvement in ranking joint mAP establishes new state-of-the-art performance on the WOMD interaction prediction challenge.
Takeaways & Limitations
MotionLM captures multimodal joint futures and supports marginal, joint, and conditional behavior prediction through one sampling-based framework.
Takeaways & Limitations
Acausal conditioning can obtain better quantitative scores largely from predictions that are nonsensical as reactions to the query agent.

Abstract

from arXiv · show

Reliable forecasting of the future behavior of road agents is a critical component to safe planning in autonomous vehicles. Here, we represent continuous trajectories as sequences of discrete motion tokens and cast multi-agent motion prediction as a language modeling task over this domain. Our model, MotionLM, provides several advantages: First, it does not require anchors or explicit latent variable optimization to learn multimodal distributions. Instead, we leverage a single standard language modeling objective, maximizing the average log probability over sequence tokens. Second, our approach bypasses post-hoc interaction heuristics where individual agent trajectory generation is conducted prior to interactive scoring. Instead, MotionLM produces joint distributions over interactive agent futures in a single autoregressive decoding process. In addition, the model's sequential factorization enables temporally causal conditional rollouts. The proposed approach establishes new state-of-the-art performance for multi-agent motion prediction on the Waymo Open Motion Dataset, ranking 1st on the interactive challenge leaderboard.

1. Introduction

MotionLM frames multi-agent motion forecasting as autoregressive language modeling over discrete motion tokens, jointly generating interactive futures with temporal causality. The approach targets inconsistent independent forecasts and reports state-of-the-art interactive performance on WOMD.

Independent per-agent forecasts miss future dependencies between agents, producing inconsistent scene-level predictions for planning.
MotionLM jointly generates interacting agents’ trajectories through a single temporally causal decoding process over discrete motion tokens.The model maximizes the log probability of token sequences and samples interacting agents’ tokens step-by-step.
MotionLM learns multimodality through sampling without anchors or explicit latent-variable optimization, while combining trajectory generation and interaction modeling.
6% improvement in ranking joint mAP establishes new state-of-the-art performance on the WOMD interaction prediction challenge.
The paper analyzes temporally causal conditional predictions, which are largely unsupported by current joint forecasting models.

2. Related work

Related work spans marginal, interactive, and autoregressive trajectory prediction, with prior methods using anchors, latent variables, mixtures, heuristics, or beam search. MotionLM instead samples discrete motion-token sequences to generate multimodal joint trajectories directly.

Marginal trajectory prediction: Marginal predictors model individual-agent futures but omit interactions among agents.Some address multimodality with predefined anchors or intention points, while others learn diverse modes with latent variables.
Interactive trajectory prediction: Interactive prediction models joint futures, but the task has been less studied than marginal prediction on WOMD.The cited leaderboard comparison reports 71 published marginal-prediction entries versus 14 interaction-prediction entries.
Interactive trajectory prediction: Prior interactive approaches include joint mixture models, pairwise distributions, conditional predictions, and products of marginal probabilities.
Autoregressive trajectory prediction: Autoregressive predictors produce scene-consistent multi-agent trajectories, commonly using latent-variable models, Gaussian mixtures, or beam search.
Autoregressive trajectory prediction: MotionLM directly samples multimodal joint trajectories from learned discrete motion-token distributions without latent variables or beam search.
Discrete sequence modeling in continuous domains: Discrete sequence modeling converts continuous-domain outputs into categorical predictions at each step, as demonstrated in image and audio generation.

3. MotionLM

MotionLM models multi-agent futures as joint sequences of discrete motion tokens, using a temporally causal autoregressive decoder to capture multimodality and agent interactions. Its factorization supports conditional rollouts while preserving temporal dependencies.

Joint probabilistic rollouts: The model factorizes joint future action sequences into conditionals over prior actions and scene context, treating agents as conditionally independent within each timestep.The paper reports that 2 Hz reactions are sufficient to surpass prior state-of-the-art joint prediction models.
Joint trajectory decoder: Multimodality arises from categorical token sampling, without anchors or explicit latent-variable optimization.The discrete-token formulation turns per-timestep sampling into classification with a standard softmax.
Training objective: Training maximizes likelihood over observed multi-agent action sequences using teacher forcing with ground-truth prefixes for all target agents.Teacher forcing avoids sampling during training and supports temporal parallelization in attention-based architectures.
Model architecture: The scene encoder integrates roadgraph, traffic-light, and agent-history modalities, while the trajectory decoder uses cross-attention to scene features and self-attention over agent motion tokens.The encoder follows an early-fusion design, and the decoder generates motion-token sequences for multiple agents.
Joint trajectory decoder: MotionLM represents continuous trajectories as discrete motion-token sequences and generates multiple agents’ actions jointly through autoregressive decoding.The decoder samples tokens step by step, with interacting agents attending to one another.
Enforcing temporal causality: Temporally causal conditioning fixes one query agent’s action sequence and rolls out the other agents, avoiding certain spurious correlations caused by violating temporal order.The model preserves dependencies in which parents occur earlier and children later in time.

4. Experiments

MotionLM is evaluated on marginal and interactive WOMD forecasting, with ablations of interactive attention, rollout count, and conditional prediction. It achieves strong marginal performance, state-of-the-art interactive performance, and more scene-consistent predictions when agents interact during rollouts.

Evaluation setup: Experiments evaluate marginal and joint forecasting on WOMD, whose interactive challenge requires six weighted joint trajectories for two interacting agents.The evaluation uses soft mAP, mAP, miss rate, minADE, minFDE, and a custom scene-level overlap metric.
Marginal motion prediction: MotionLM ranks second in marginal soft mAP, less than 2% behind MTRA, while reducing miss rate by 9% relative to the previous state of the art.Its autoregressive rollouts capture multimodal future behavior without trajectory anchors or static intention points.
Interactive motion prediction: 6% relative mAP and 3% relative miss-rate improvements over JFP establish state-of-the-art interactive WOMD performance.Unlike JFP, MotionLM generates joint rollouts directly rather than scoring pairs of previously constructed marginal trajectories.
Interactive attention frequency: More frequent interactive attention generally improves prediction metrics and reduces implausible overlaps between agents’ predicted trajectories.With zero interactive attention, agents can produce scene-inconsistent overlaps; at 2 Hz, they attend to each other 16 times during the 8-second rollout.
Number of rollouts: Additional rollouts generally improve performance; 32 rollouts surpass the previous top joint-mAP entry, while final leaderboard results use 512 rollouts per replica.Rollout aggregation clusters samples to six modes for evaluation.
Conditional rollouts: Acausal conditioning improves soft mAP by 8.2% versus 3.7% for temporally causal conditioning relative to marginal predictions.Although acausal conditioning scores better, its predictions can be nonsensical as reactions because future information is exposed in the wrong direction.

5. Conclusion and future work

MotionLM forecasts interactive, multimodal futures through multi-agent rollouts over discrete motion tokens and achieves state-of-the-art WOMD interaction performance. The model also supports conditional rollouts that can reflect another agent’s ground-truth behavior.

MotionLM captures joint distributions over multimodal futures using multi-agent rollouts over discrete motion tokens.
The model establishes new state-of-the-art performance on the WOMD interactive prediction challenge.
Future work includes integrating multi-agent rollouts with model-based planning and distilling large autoregressive teachers into faster student models.
Conditioning on a vehicle’s ground-truth turn changes the pedestrian prediction from crossing the road to yielding.

A. Motion token vocabulary

MotionLM discretizes delta actions into a compact motion-token vocabulary and generates 16 tokens per agent for the 8-second WOMD prediction horizon.

The delta action space uses 2 Hz sampling, intervals from -18.0 m to 18.0 m, and 128 bins.
At 2 Hz, the maximum 18 m delta covers axis-aligned speeds up to 36 m/s, exceeding 99% of WOMD.
The Verlet wrapper reduces each coordinate to 13 bins, producing 169 total discrete motion tokens.
For 8-second futures, the model outputs 16 motion tokens per agent, yielding 32 flattened tokens for two-agent interactions.

B. Implementation details

The implementation uses an early-fusion scene encoder, a causal transformer decoder, teacher-forced likelihood training, and nucleus sampling for diverse motion-token generation.

Scene encoder: The scene encoding backbone follows an early-fusion network with four layers, 1024-dimensional feed-forward intermediates, four attention heads, and 92 latent queries.
Motion decoder: A causal transformer decoder autoregressively predicts motion tokens from token queries conditioned on scene encodings.
Motion decoder: The decoder uses four layers, 1024-dimensional feed-forward intermediates, and four attention heads.
Motion decoder: Masked causal attention lets agents attend to each other’s previous motion tokens while excluding future tokens.
Training: Training maximizes the likelihood of ground-truth motion-token sequences using teacher forcing for 600000 steps with a linearly decayed learning rate from 0.0006 to 0.0.
Inference: Nucleus sampling with top-p 0.95 improves sample quality while maintaining diversity.

C.1. WOMD metrics

WOMD evaluation averages metrics across three prediction horizons and object types, using marginal and joint measures that assess detection quality, misses, displacement error, and scene consistency.

All WOMD metrics are evaluated at 3, 5, and 8 seconds and averaged across object types, with up to 6 trajectories generated per target agent.
mAP evaluates precision across eight intent classes, while soft mAP does not penalize additional matching predictions as false positives.
Miss rate measures the fraction of scenarios without any prediction within the specified lateral and longitudinal error thresholds.
minADE averages Euclidean error across timesteps for the closest prediction, whereas minFDE measures error only at the final timestep.
Prediction overlap rate measures the dataset-average fraction of scenes in which predicted agents’ bounding boxes collide at any timestep.

D. Additional evaluation

Additional evaluation examines ablations, model scaling, inference latency, and qualitative prediction examples. The scaling study finds underfitting and overfitting at the smallest and largest tested model sizes, respectively.

Ablations: Ablations vary interactive attention frequency and rollout count, evaluating both ensembled and single-replica joint prediction performance.Standard deviations are computed over 8 independently trained replicas.
Scaling analysis: 27M parameters overfits, 300K underfits, and both 1M and 9M models perform decently on the WOMD interactive split.Model size varies through proportional changes in encoder and decoder layers, hidden size, and attention heads.
Latency analysis: Inference latency is measured on current-generation GPUs for different rollout counts using a single-replica joint model rolling out two agents.The latency study reports measurements across rollout settings.
Qualitative examples: Supplementary GIFs provide greatest-probability prediction examples across various driving scenes.Each example is identified by a scene ID shared with its corresponding GIF filename.

E.1. Marginal vs. Joint

Qualitative comparisons show joint predictions avoiding collisions that occur in marginal predictions across turning, crossing, lane-changing, and cyclist interactions. The section also reports scene-level joint evaluation across attention frequencies, rollout counts, and model sizes, plus inference latency.

Qualitative comparison: Joint predictions avoid collisions in examples where marginal predictions produce collisions during turning, crossing, or lane-changing interactions.The examples include vehicles yielding, adjacent vehicles slowing, and cyclists waiting or remaining in lane.
Quantitative evaluation: Table 5 evaluates scene-level joint metrics across interactive attention frequencies on the WOMD interactive validation set.Single-replica metrics include standard deviations across 8 replicas.
Quantitative evaluation: Table 6 evaluates scene-level joint metrics across numbers of rollouts per replica on the WOMD interactive validation set.Single-replica metrics include standard deviations across 8 replicas.
Quantitative evaluation: Table 7 reports joint mAP and miss rate across model sizes for 256 rollouts on the WOMD interactive validation set.The results use a single model replica except for the 9M model, which reports the mean of 8 replicas.
Latency analysis: Table 8 reports mean and standard deviation of joint-model inference latency for different rollout counts on current-generation GPUs.The table summarizes latency measurements for each rollout setting.

E.2. Marginal vs. Conditional

Conditional predictions alter non-query agents in response to fixed query-agent trajectories, with examples showing more appropriate reactions than marginal predictions. The temporally causal versus acausal comparison highlights the importance of influence direction.

Terminology: “Conditional” denotes temporally causal conditioning in the paper’s terminology.The term is used for rollouts conditioned on a query agent’s trajectory.
Conditional examples: Conditioning on turning or stationary lead-vehicle trajectories changes pedestrian and modeled-vehicle predictions from collision-prone behavior to waiting or stopping.The examples show a pedestrian remaining stationary and a vehicle making an appropriate stop.
Conditional examples: Conditioning on another vehicle’s progress changes modeled-agent behavior, producing more progress during a turn and acceleration behind a merging vehicle.These examples use the query vehicle’s turn or merge progress as the conditioning trajectory.
Temporally causal versus acausal conditioning: The temporally causal model ignores a trailing agent’s deceleration when predicting the lead vehicle but predicts the trailing vehicle stopping after a lead vehicle decelerates.The acausal model incorrectly lets the lead vehicle stop in response to the trailing vehicle’s deceleration.