Source-linked AI summary
MotionDiffuser: Controllable Multi-Agent Motion Prediction using Diffusion
Chiyu Max Jiang, Andre Cornman, Cheolho Park, Ben Sapp, Yin Zhou, Dragomir Anguelov
TL;DR
Motion prediction requires diverse, jointly consistent futures for multiple interacting agents under scene context. MotionDiffuser uses conditional diffusion with a permutation-invariant set denoiser, PCA-based trajectory representations, and differentiable constrained sampling. It reports state-of-the-art multi-agent motion-prediction results on the Waymo Open Motion Dataset.
Problem
Autonomous-vehicle motion prediction must capture multimodal future outcomes and jointly reason about interacting agents rather than independently sampling conflicting marginals.
Method
MotionDiffuser learns a permutation-invariant joint trajectory distribution with conditional diffusion, PCA-based latent representations, and controlled sampling through arbitrary differentiable trajectory costs.
Results
MotionDiffuser demonstrates state-of-the-art multi-agent motion-prediction results on the Waymo Open Motion Dataset.
Takeaways & Limitations
The framework supports diverse multimodal joint futures and controlled trajectory generation for rules, physical constraints, in-painting, and tailored simulation scenarios.
Takeaways & Limitations
Future work includes applying the diffusion-based generative modeling technique to planning and scene generation in autonomous vehicles.
Abstract
from arXiv · showhide
We present MotionDiffuser, a diffusion based representation for the joint distribution of future trajectories over multiple agents. Such representation has several key advantages: first, our model learns a highly multimodal distribution that captures diverse future outcomes. Second, the simple predictor design requires only a single L2 loss training objective, and does not depend on trajectory anchors. Third, our model is capable of learning the joint distribution for the motion of multiple agents in a permutation-invariant manner. Furthermore, we utilize a compressed trajectory representation via PCA, which improves model performance and allows for efficient computation of the exact sample log probability. Subsequently, we propose a general constrained sampling framework that enables controlled trajectory sampling based on differentiable cost functions. This strategy enables a host of applications such as enforcing rules and physical priors, or creating tailored simulation scenarios. MotionDiffuser can be combined with existing backbone architectures to achieve top motion forecasting results. We obtain state-of-the-art results for multi-agent motion prediction on the Waymo Open Motion Dataset.
1. Introduction
MotionDiffuser addresses multimodal, interacting multi-agent futures with a permutation-invariant conditional diffusion representation. It also supports controlled sampling through differentiable constraints and improves representation with PCA-based latent diffusion and trajectory clustering.
- Challenges: Motion prediction must represent multimodal futures jointly because independently sampling agent marginals can produce unrealistic and conflicting outcomes.The problem also requires constraints over future time steps or specified behaviors for one or more agents.
- MotionDiffuser: MotionDiffuser models the joint future distribution of multiple agents with a conditional denoising diffusion model.It generates samples by iteratively refining noisy trajectories from pure Gaussian noise.
- MotionDiffuser: A cross-attention-based permutation-invariant denoiser learns the motion distribution regardless of agent ordering.The design targets joint reasoning over a set of interacting agents.
- Representation enhancements: PCA-based latent trajectory diffusion and improved trajectory sample clustering are proposed to further boost model performance.The PCA-based representation is one of the highlighted enhancements to MotionDiffuser.
- Controlled sampling: Controlled trajectory sampling uses arbitrary differentiable cost functions to enforce rules, controls, trajectory in-painting, and tailored simulation scenarios.These applications extend diffusion-based generation beyond unconstrained future sampling.
2. Related Work
Prior motion-prediction work includes supervised, generative, graphical-model, and joint multi-agent approaches. MotionDiffuser is positioned among emerging diffusion methods and claims to be the first to model multi-agent joint motion distributions with diffusion models.
- Constrained sampling: Diffusion models have been applied to controllable inverse problems such as image in-painting, colorization, and sparse-view computed tomography.The related-work discussion connects controllable sampling with conditioning generation on information unavailable during training.
- Motion prediction: Supervised motion-prediction methods use logged trajectories and losses such as L2, while multimodality has motivated anchors, goals, occupancy heatmaps, and trajectory decoding.Examples include MultiPath, MultiPath++, Wayformer, SceneTransformer, DenseTNT, Home, and GoHome.
- Generative models: Generative approaches model future trajectories as conditional probability inference using GANs, C-VAEs, normalizing flows, and emerging diffusion models.The conditional form is p(s; c), where s denotes future trajectories and c denotes context or observation.
- Positioning: MotionDiffuser states that it is the first work to use diffusion models for the multi-agent joint motion distribution.The paper distinguishes this scope from earlier diffusion work on human motion, planning, and pedestrian uncertainty.
- Multi-agent motion prediction: Multi-agent alternatives include fixed-set joint predictions, conditional models based on controlled-agent motions, and probabilistic graphical models with static or dynamic interaction graphs.Examples include SceneTransformer, M2I, WIMP, PIP, CBP, DSDNet, MFP, JFP, and RAIN.
3. Method
MotionDiffuser models joint, scene-conditional future trajectories for multiple agents with diffusion, using permutation-equivariant attention, compressed PCA representations, exact log-probability inference, and constrained sampling.
- Diffusion model preliminaries: Diffusion models learn trajectory distributions by denoising noisy samples across noise levels and can generate samples by iteratively refining Gaussian noise.The model follows ODE-based sampling with a linear noise schedule and 32 Huen integration steps.
- Diffusion model preliminaries: Conditional diffusion models represent future trajectories x for an agent set conditioned on scene context c using a conditional denoiser and score function.The context includes scene information encoded for the trajectory prediction task.
- Multi-agent trajectory model: The set denoiser uses repeated self-attention across agents and cross-attention to per-agent context tokens, while omitting agent positional encoding to preserve permutation equivariance.Self-attention captures joint multi-agent structure, and cross-attention incorporates scene context and noise level information.
- Exact log probability inference: Exact sample log probabilities are computed by integrating the instantaneous change-of-variables equation, with Jacobian-trace computation costing O(n^2) in trajectory dimensionality.PCA reduces n, while Hutchinson’s estimator can reduce trace computation to O(n).
- Constrained trajectory sampling: Differentiable trajectory costs guide sampling from the learned distribution toward constrained outcomes, including attractors for target locations and repellers that discourage close interactions.The framework supports physical rules, collision avoidance, trajectory in-painting, and custom traffic scenarios.
- Trajectory representation enhancements: PCA compresses smooth trajectories: three components explain 99.7% of variance for 80 × 2 trajectories, while ten components are used for more accurate reconstruction.The compressed representation is reported to improve inference speed, controlled-trajectory success, accuracy, and performance.
4. Experiment and Results
Experiments evaluate MotionDiffuser’s PCA representation, multi-agent prediction, and controllable sampling on Waymo Open Motion Dataset splits. The method achieves state-of-the-art prediction results and produces constrained trajectories while preserving realism.
- PCA Representation: Using 10 principal components yields 0.06 meters average reconstruction error per waypoint.The PCA analysis examines randomly selected trajectories from the Waymo Open Dataset training split.
- PCA Representation: Higher-order PCA components mainly encode high-frequency information with small impact on the final trajectory.These components increasingly resemble one another and deviate only slightly from the dataset mean.
- Multi-Agent Motion Prediction: MotionDiffuser achieves state-of-the-art results on the Waymo Open Dataset Interactive split.The evaluation uses joint metrics averaged across object types and time horizons of 3, 5, and 8 seconds.
- Multi-Agent Motion Prediction: Compared with JFP on the test split, MotionDiffuser improves minSADE and minSFDE while slightly underperforming on mAP and Overlap.It outperforms all other methods on mAP and Overlap besides JFP.
- Controllable Trajectory Synthesis: The proposed constrained sampler maintains high realism while enforcing trajectories more effectively than unconstrained sampling, though optimization enforces constraints most strongly.Direct test-time optimization enforces constraints but produces unrealistic trajectories, while the proposed method remains near the data manifold.
- Controllable Trajectory Synthesis: A repeller constraint with radius 5m decreases overlap between joint predictions by an order of magnitude.The experiment applies the constraint between all pairs of modeled agents.
5. Ablation Studies
Ablations show that score thresholding, PCA compression, Transformer denoising, and agent-level self-attention are important components of MotionDiffuser’s performance and joint consistency.
- Constraint Sampling: Removing Score Thresholding results in significantly worse constraint satisfaction.The ablation compares Ours(-ST) with the full method in the controllable sampling experiments.
- Representation: Using an uncompressed trajectory representation instead of PCA degrades performance significantly.This ablation is reported on the WOMD Interactive Validation Split.
- Denoiser Architecture: Replacing the Transformer with a simple MLP reduces performance.The comparison isolates the denoiser architecture while evaluating the same overall framework.
- Denoiser Architecture: Removing denoiser self-attention reduces performance, indicating that attention between agents’ noisy future trajectories supports consistent joint predictions.Cross-attention remains, preserving conditioning on scene context and noise level.
6. Conclusion and Discussions
MotionDiffuser models diverse multimodal joint futures for multiple agents with a permutation-invariant Transformer denoiser and constrained sampling framework. The paper reports state-of-the-art Waymo results and demonstrates attractor and repeller constraints.
- Conclusion: MotionDiffuser learns a diverse, multimodal joint future distribution for multiple agents.The framework is diffusion-model based and targets joint rather than independent agent futures.
- Conclusion: Its Transformer-based set denoiser is permutation invariant across agents.The architecture is designed to represent agent sets without dependence on their ordering.
- Conclusion: The constrained sampling framework demonstrates attractor and repeller constraints for controlled trajectory generation.The conclusion describes the framework as general and flexible.
- Conclusion: The paper demonstrates state-of-the-art multi-agent prediction results and effective trajectory control on the Waymo Open Motion Dataset.Future work considers applying diffusion-based generative modeling to planning and scene generation.
Waymo LLC
The document includes an arXiv version identifier dated 5 June 2023.
- The paper is identified as arXiv:2306.03083v1, dated 5 Jun 2023.
2. Implementation Details
MotionDiffuser is trained on the Waymo Open Motion Dataset with a Wayformer encoder backbone and transformer denoiser. Training and inference use different numbers of noise vectors.
- MotionDiffuser is trained on the Waymo Open Motion Dataset for 2 × 10^6 training steps using 32 TPU shards.The optimizer is AdamW with weight decay 0.03, a 5 × 10^-4 learning rate, 10^4 warmup steps, and linear decay.
- The model uses a Wayformer encoder with 128 latent embeddings, each having hidden size 256.
- The transformer denoiser contains four self-attention and cross-attention layers with hidden size 256 and intermediate size 1024.ReLU activations are used throughout, and the noise level is embedded with 128 random Fourier features.
- MotionDiffuser denoises 128 random noise vectors during training and 256 during inference before clustering.
3. Network Preconditioning
MotionDiffuser uses network preconditioning to construct its denoiser from scaled skip, input, output, and noise-level components. The experiments fix σdata at 0.5.
- The denoiser combines a scaled skip connection with a neural network applied to a scaled input and conditioned on context and noise level.Its form is Dθ(x; c, σ) = cskip(σ)x + cout(σ)Fθ(cin(σ)x; c, cnoise(σ)).
- cin(σ) scales the network input so that inputs to Fθ have unit variance.
- cskip(σ) modulates the skip connection in the preconditioned denoiser.
- cout(σ) modulates the network output, while cnoise(σ) scales the noise level supplied to the network.
- All experiments use σdata = 0.5.
4. Inference Latency
The paper evaluates inference latency as the number of sampling steps varies, using a single V100 GPU with batch size one. Table 1 reports latency alongside quality on the WOMD Interactive Validation Split.
- Inference latency is measured over varying numbers of sampling steps T using a single V100 GPU and batch size 1.
- Table 1 compares model inference latency and quality on the WOMD Interactive Validation Split.