Source-linked AI summary
MultiPath++: Efficient Information Fusion and Trajectory Aggregation for Behavior Prediction
Balakrishnan Varadarajan, Ahmed Hefny, Avikalp Srivastava, Khaled S. Refaat, Nigamaa Nayakanti, Andre Cornman, Kan Chen, Bertrand Douillard, Chi Pang Lam, Dragomir Anguelov, Benjamin Sapp
TL;DR
MultiPath++ addresses the challenge of forecasting highly multimodal road-user behavior by revising how heterogeneous scene information and future distributions are represented and combined. Its sparse encoding, context-aware fusion, learned anchors, and output aggregation achieve state-of-the-art results on popular behavior-prediction benchmarks.
Problem
Forecasting vehicles, cyclists, and pedestrians is critical for safe, comfortable, and human-like autonomous driving, but autonomous-driving behavior prediction must capture heterogeneous scene information and multimodal futures.
Method
MultiPath++ uses sparse polylines and raw agent states, multi-context gating for context-aware fusion, end-to-end learned anchor embeddings, and ensemble output aggregation for multimodal predictions.
Results
MultiPath++ achieves state-of-the-art performance, ranking 1st on the Waymo Open Motion Dataset leaderboard and 4th in the Argoverse Motion Forecasting Competition as of November 1, 2021.
Takeaways & Limitations
The paper offers an empirically validated reference set of design choices emphasizing sparse encoding, efficient fusion methods, learned anchors, and practical training and inference techniques.
Takeaways & Limitations
Pre-defined anchor trajectories can fit scenes poorly and require many modes plus a two-phase learning process, motivating learned anchors.
Abstract
from arXiv · showhide
Predicting the future behavior of road users is one of the most challenging and important problems in autonomous driving. Applying deep learning to this problem requires fusing heterogeneous world state in the form of rich perception signals and map information, and inferring highly multi-modal distributions over possible futures. In this paper, we present MultiPath++, a future prediction model that achieves state-of-the-art performance on popular benchmarks. MultiPath++ improves the MultiPath architecture by revisiting many design choices. The first key design difference is a departure from dense image-based encoding of the input world state in favor of a sparse encoding of heterogeneous scene elements: MultiPath++ consumes compact and efficient polylines to describe road features, and raw agent state information directly (e.g., position, velocity, acceleration). We propose a context-aware fusion of these elements and develop a reusable multi-context gating fusion component. Second, we reconsider the choice of pre-defined, static anchors, and develop a way to learn latent anchor embeddings end-to-end in the model. Lastly, we explore ensembling and output aggregation techniques -- common in other ML domains -- and find effective variants for our probabilistic multimodal output representation. We perform an extensive ablation on these design choices, and show that our proposed model achieves state-of-the-art performance on the Argoverse Motion Forecasting Competition and the Waymo Open Dataset Motion Prediction Challenge.
1 Introduction
MultiPath++ addresses multimodal behavior prediction by revisiting how heterogeneous scene inputs, interactions, trajectories, anchors, and ensemble outputs are represented and combined. The resulting design choices achieve leading benchmark rankings and are supported by ablation studies.
- Motivation: Behavior prediction must represent stochastic, highly multimodal futures because other agents’ intentions are unobservable.Possible outcomes can include distinct maneuvers such as turning left or right at an intersection.
- Motivation: Driving scenes combine heterogeneous, interrelated road and agent inputs, while explicit joint distributions over many agents scale exponentially.Relevant inputs include road geometry, traffic signals, and agent motion history.
- Input representation: MultiPath++ replaces rasterized CNN encoding with sparse polylines, raw agent states, and element-specific recurrent interaction encodings.Its compute complexity scales with the number of scene elements rather than spatial-grid size.
- Context fusion: Multi-context gating conditions groups of road and agent elements on summary context vectors, providing an efficient cross-attention-like fusion mechanism.The efficiency/quality trade-off depends on the context-vector size.
- Trajectory modeling: The model learns latent anchor embeddings end-to-end and compares trajectory representations based on kinematic controls and continuous-time polynomials.Learned anchors outperform the original static anchors while simplifying model creation to one step.
- Ensembling: Ensembling and output aggregation provide additional gains despite mixture components lacking correspondence across predictor heads.The method also allows heads to predict richer distributions with L > M modes when benchmarks require M trajectories.
- Results: As of November 1, 2021, MultiPath++ ranks 1st on the Waymo Open Motion Dataset leaderboard and 4th on the Argoverse Motion Forecasting Competition.The paper presents these choices as empirically validated through ablation studies.
2 Related Work
Prior behavior-prediction work differs across representations for roads, motion history, interactions, trajectories, and output distributions. The literature contrasts convenient rasterized encodings with compact polylines and several alternatives for generating diverse futures.
- Road encoding: Road encoding divides between rasterized top-down images and sparse polylines describing curves such as lanes, crosswalks, and boundaries.Rasterization unifies inputs for CNNs but can constrain field of view, long-range interactions, and continuous-state representation.
- Motion history: Motion history is commonly encoded with recurrent networks or temporal convolutions, while raster methods render state sequences as image masks.The rendered representation may separate time information into an additional image channel.
- Agent interactions: Agent interactions are often modeled by permutation-invariant pooling or attention over neighboring agents, whereas raster methods capture them through spatial convolutions.Some approaches instead jointly roll out agent policies step by step.
- Trajectory decoding: Trajectory decoders commonly produce state sequences or differences, with alternatives including Verlet integration, vehicle controls, and continuous polynomial representations.Other methods predict heatmaps before sampling or enumerate trajectories using heuristic cost evaluation.
- Output space: Nearly all work assumes independent per-agent outputs, while a smaller set models joint interactions asymmetrically or symmetrically.The cited exceptions explicitly represent interactions in the output space.
- Trajectory distributions: Gaussian Mixture Models are popular because their compact parameterization can represent diverse possible trajectories.Mode collapse is addressed through training techniques or trajectory anchors.
3 Model Architecture
MultiPath++ encodes heterogeneous driving scenes as sparse modality-specific representations and fuses them with multi-context gating. It also replaces static trajectory anchors with trainable latent embeddings and uses context-aware prediction components.
- Prediction Architecture: The predictor outputs a Gaussian Mixture Model for future positions, with MCG producing one embedding per trajectory for each agent.The architecture contains an encoding step followed by a predictor head conditioned on anchors.
- Input Representation: MultiPath++ processes each input modality with separate encoders, then fuses their compact representations using multi-context gating.The model includes agent history, road-network, neighboring-agent, and AV-relative features.
- Input Representation: Agent state history is transformed into an agent-centric frame, while road elements are represented as piecewise-linear polylines.The latest agent pose is placed at the origin with heading east; the closest P = 128 polylines are transformed into each agent’s frame.
- Multi-Context Gating: MCG assigns updated embeddings to set elements and computes a context vector while preserving permutation-equivariance and permutation-invariance.The set size can vary across calls, supporting unordered road elements and agent relationships.
- Multi-Context Gating: Compared with cross-attention, context gating summarizes one input set with a single context vector, trading representational power for computational efficiency when dc < md.Stacked CG blocks use running-average skip-connections, and MCG is applied throughout the architecture.
- Prediction Architecture: Learned anchor embeddings are trained end-to-end and correspond one-to-one with output trajectory modes, replacing input-independent static anchors.The embeddings are trainable model parameters independent of the input.
4 Ensembling predictor heads via bootstrap aggregation
MultiPath++ ensembles predictor heads with bootstrap aggregation and aggregates their Gaussian-mixture outputs into a compact distribution. The aggregation explicitly favors trajectory diversity under benchmark-relevant criteria.
- Bootstrap Aggregation: Bootstrap aggregation trains E predictor heads with random initialization and updates each head independently with 50% probability per example.The procedure is intended to encourage complementary information across heads.
- Output Aggregation: Because mixture components lack correspondence across heads, MultiPath++ combines all head outputs and compresses them into an M-mode Gaussian Mixture Model.Each head may predict L > M modes, while the task requires M modes for benchmark metrics.
- Output Aggregation: The aggregation fits the combined mixture with an iterative hard-assignment clustering algorithm that can be trained end-to-end as a final network layer.The method resembles Expectation-Maximization but uses hard cluster membership.
- Output Aggregation: M cluster centroids are selected greedily to maximize the probability that a sampled centroid lies within distance τ of a selected centroid.This criterion directly optimizes trajectory diversity and is suited to miss rate, mAP, and minADE.
5 Experiments
The experiments evaluate MultiPath++ on the Waymo Open Motion Dataset and Argoverse using dataset-specific competition metrics. On the reported leaderboards, MultiPath++ ranks first on Waymo and fourth on Argoverse.
- Datasets: The Waymo Open Motion Dataset contains 1.1M examples from 103K 20-second scenarios, with 1 second of history and 8 seconds of future resampled at 5Hz.Examples are derived from real-world urban and suburban driving and include rich agent-state attributes.
- Datasets: Argoverse contains 333K scenarios with trajectory histories, context agents, and lane-centerline inputs, sampled at 10Hz over a 3-second prediction horizon.The dataset provides 2 seconds of past history.
- Metrics: The evaluation reports competition-specific metrics, including minimum distance errors, miss rate, mAP, and overlap rate.The definitions and leaderboard conventions vary by dataset, including a distinct WOMD miss-rate definition.
- Results: Rank 4, 1.793 brier-minDE, 1.214 minFDE, 0.132 MR, and 0.790 minADE are reported for MultiPath++ on the Argoverse leaderboard.The table compares MultiPath++ with published state-of-the-art methods under k = 6, d = 2m, and t = 3s.
5.3 MultiPath baseline
The reference MultiPath baseline preserves the original top-down rasterized input and backbone with a simpler splat-rendering implementation. It uses a 400 × 400 grid, ResNet18, and 128 shared static anchors.
- Baseline Architecture: The reference baseline keeps MultiPath’s top-down scene representation and backbone while using sparse splat rendering for implementation efficiency.Scene elements are sampled uniformly, orthographically projected, and rendered without anti-aliasing in the compute graph.
- Baseline Architecture: The baseline uses a 400 × 400 grid of 0.2m × 0.2m cells, giving an 80m field of view centered on the sensing vehicle in WOMD.The backbone is ResNet18.
- Baseline Architecture: The baseline uses 128 static k-means anchors shared across vehicles, pedestrians, and cyclists.The shared anchor set is used for simplicity.
5.4 External benchmark results
MultiPath++ performs strongly on external motion-prediction benchmarks, ranking first on Waymo and achieving top-five performance on most Argoverse metrics. Qualitative examples show predictions handling varied road layouts and agent interactions.
- Top-5 performance on most Argoverse metrics, with MultiPath++ ranked first on every Waymo metric.
- More recent WOMD methods outperform original MultiPath, while sparse-representation methods outrank the best CNN-based top-down model.
- MultiPath++ handles different road layouts and agent interactions in qualitative WOMD and Argoverse examples.
5.6 Ablation Study
The ablation study evaluates architecture choices for set functions, trajectory representations, anchors, ensembling, and qualitative multimodal predictions. Results favor context-aware set-function modeling, learned anchors, and controls or raw coordinates depending on the metric and feasibility requirement.
- Ablation Study: Ablations compare architecture choices and assess how they affect model performance.
- Qualitative predictions: Qualitative WOMD examples depict multimodal interaction outcomes, including yielding, turning, and bimodal pedestrian predictions.
- Qualitative predictions: Qualitative Argoverse examples show predictions following different lane geometries.
- Set functions: Multi-context gating represents invariant encoders and equivariant trajectory-producing set functions using contextual gating between elements.
- Trajectory representation: Polynomial trajectory representations hurt performance, while controls-based outputs perform better than polynomials on distance metrics.
- Trajectory representation: Raw-coordinate trajectories perform best for distance-based metrics but have a non-trivial rate of kinematic infeasibility.
- Ensembling: Ensembling produces an over-complete trajectory set that is aggregated to the six trajectories required for WOMD submission.
- Anchor representation: The study explores both learned and k-means-based anchor representations.
5.7 Discussion
MultiPath++ improves over MultiPath through sparse, agent-centric modeling and several validated design choices. Learned anchors and aggregation/ensembling target different trajectory-quality requirements, including diversity and average-distance metrics.
- Overall comparison: MultiPath++ is a significant improvement over MultiPath, primarily changing dense raster inputs to sparse, element-based representations with agent-centric coordinates.The discussion states that other design choices are validated separately.
- Fusion architecture: A five-layer multi-context gating component achieves the best performance among the tested fusion depths.Increasing MCG depth consistently improves results, while MLP+MaxPool performs worst among the tested set-function variants.
- Anchor design: Learned anchors outperform anchors obtained a priori via k-means.The authors suggest richer inputs, improved architecture, and larger batch sizes may explain the difference from original MultiPath findings.
- Ensembling and aggregation: Ensembling improves most metrics, whereas aggregation of 64 trajectories particularly improves miss rate and AUC but slightly worsens average-distance metrics.The experiments indicate aggregation is suited to preserving diversity, while straight-up ensembling better captures the average distribution.
- Ensembling and aggregation: Aggregation can improve trajectory diversity, remove unrealistic trajectories, and enable predictions spanning multiple lanes.These effects are illustrated qualitatively in Figures 7–9.
5.8 Conclusion
MultiPath++ combines redesigned input encoding, fusion, and output-distribution choices into a behavior prediction system. The paper reports state-of-the-art benchmark results and practical guidance from empirical analysis.
- Conclusion: MultiPath++ combines sparse input encoding, efficient fusion, and output-distribution representations in a behavior prediction system.The paper also analyzes these choices empirically and surveys existing methods.
- Conclusion: The paper reports state-of-the-art results on popular behavior-prediction benchmarks.It highlights sparse encoding, efficient fusion, control-based methods, and learned anchors.
- Conclusion: The paper provides practical guidance for training and inference to improve robustness, increase diversity, handle missing data, and accelerate convergence.
A Details and Derivation of Aggregation Algorithm
The aggregation algorithm compresses an overcomplete multimodal trajectory distribution into a smaller GMM while addressing aleatoric and epistemic uncertainty. It uses an EM-like local optimization procedure with approximation and greedy initialization choices.
- Motivation and uncertainty: Overcomplete trajectory representations followed by aggregation address aleatoric and epistemic uncertainty in future behavior predictions.Aleatoric uncertainty concerns natural outcome variation, while epistemic uncertainty concerns variation across model outputs.
- Motivation and uncertainty: The overcomplete representation encourages diverse trajectories and greedy aggregation preserves diversity in the final output.Ensembling is used to reduce variation across model outputs associated with epistemic uncertainty.
- Aggregation objective: The aggregation objective obtains an M-mode GMM that minimizes DKL(Ψ||¯Ψ), equivalently maximizing expected log likelihood under the overcomplete distribution.
- Optimization: Because direct optimization is intractable, the method uses an Expectation-Maximization-like algorithm to seek a local maximum.The hidden variable h denotes a mixture component in the compact representation, and the Q function supports parameter updates.
- Approximation: The method approximates posterior expectations using overcomplete cluster means, an assumption that becomes more focused as within-cluster variance decreases.
- Initialization: GMM centroid initialization maximizes coverage within distance τ, but exact subset optimization is NP-hard, so an M-sized subset is selected greedily.The greedy selection is supported by the submodularity guarantee described in the text.
B Multipath baseline system diagram
The baseline MultiPath system uses a rasterized input representation together with its backbone and output architecture. Figure 10 provides the corresponding system diagram.
- System diagram: Figure 10 depicts MultiPath’s splat-rendering rasterization of input points, backbone, and output architecture.The figure’s detailed description is located in Section 5.3.