Source-linked AI summary
CoverNet: Multimodal Behavior Prediction using Trajectory Sets
Tung Phan-Minh, Elena Corina Grigore, Freddy A. Boulton, Oscar Beijbom, Eric M. Wolff
TL;DR
Urban trajectory prediction must represent diverse agent intents with likely, physically realizable futures, while existing approaches can collapse modes or produce unconstrained trajectories. CoverNet classifies over fixed or dynamically generated trajectory sets designed for coverage and feasibility. On real-world driving datasets, it outperforms similar state-of-the-art methods, with dynamic generation further improving performance.
Problem
Urban trajectory prediction must capture multiple possible agent behaviors and likelihoods while producing physically realizable trajectories.
Method
CoverNet frames probabilistic trajectory prediction as classification over fixed or dynamically generated trajectory sets that approximate reachable states and support coverage.
Results
CoverNet outperforms similar state-of-the-art methods on real-world self-driving datasets, including public and internal data.
Takeaways & Limitations
Trajectory-set classification can provide coverage, exclude dynamically infeasible trajectories, and avoid mode collapse while keeping trajectory sets manageable over realistic horizons.
Takeaways & Limitations
The approach assumes access to sufficiently accurate object-detection and tracking state outputs, and nuScenes may limit useful trajectory-set size.
Abstract
from arXiv · showhide
We present CoverNet, a new method for multimodal, probabilistic trajectory prediction for urban driving. Previous work has employed a variety of methods, including multimodal regression, occupancy maps, and 1-step stochastic policies. We instead frame the trajectory prediction problem as classification over a diverse set of trajectories. The size of this set remains manageable due to the limited number of distinct actions that can be taken over a reasonable prediction horizon. We structure the trajectory set to a) ensure a desired level of coverage of the state space, and b) eliminate physically impossible trajectories. By dynamically generating trajectory sets based on the agent's current state, we can further improve our method's efficiency. We demonstrate our approach on public, real-world self-driving datasets, and show that it outperforms state-of-the-art methods.
1. Introduction
CoverNet addresses multimodal urban trajectory prediction by classifying over a structured trajectory set, avoiding mode collapse while supporting coverage and physical feasibility. Its dynamic representation makes predictions consistent with the agent’s current state.
- Urban driving requires predicting multiple likely futures because agents have diverse preferences and intents.Predictions must also account for road context and recent agent behavior.
- Multimodal regression can collapse to one mode, while unconstrained position predictions may produce physically impossible trajectories.
- CoverNet formulates multimodal probabilistic prediction as classification over a trajectory set with manageable size over reasonable horizons.The approach exploits the relatively few distinct actions available over a standard six-second horizon.
- The trajectory set can be designed for state-space coverage and dynamic feasibility, addressing requirements that conventional outputs may not guarantee.
- Dynamic outputs represent motion relative to the initial state and are converted to positions through a dynamics model.This representation supports physically valid trajectories and diversity across different speeds.
- The paper introduces fixed and dynamic trajectory-set generation, evaluates CoverNet on nuScenes, and compares classification with multimodal regression.
2. Related Work
Prior trajectory-prediction methods differ in how they represent uncertainty, but face trade-offs involving multimodality, physical feasibility, computational cost, and evaluation comparability. CoverNet instead classifies directly over a structured set of possible trajectories.
- Stochastic approaches: Stochastic methods represent alternatives through sampling, but repeated rollouts or sampling requirements can make inference computationally expensive.
- Unimodal approaches: Unimodal methods output one trajectory and may average incompatible behaviors into nonsensical paths.
- Multimodal approaches: Occupancy maps capture multiple outcomes flexibly but can require large grids, lack well-defined trajectory sampling, and add inference computation.
- Multimodal approaches: Multimodal regression can suffer mode collapse, motivating fixed anchor boxes in prior work.
- Physical constraints: Most methods do not explicitly encode motion constraints, so their predicted trajectories may be physically infeasible.
- Motion planning: Unlike motion planning, CoverNet classifies over possible trajectories because other agents’ goals and preferences are unknown.The paper accepts small prediction errors over moderate three- to six-second horizons.
- Evaluation: Urban trajectory-prediction comparisons are difficult because public datasets vary in size, domain, and availability, with no standard benchmark.
3. Method
CoverNet frames multimodal trajectory prediction as classification over a designed trajectory set, using fixed or state-dependent generation to balance coverage, efficiency, and physical feasibility.
- Overview: CoverNet predicts future states by classifying over a trajectory set conditioned on agent histories, scene context, and a high-definition map.The model focuses on single-agent prediction and uses a scene representation containing map and agent information.
- Trajectory-set representation: The trajectory-set formulation uses classification to represent multiple probabilistic behaviors while avoiding mode collapse and allowing explicit coverage and feasibility requirements.The approach exploits the relatively small number of distinct actions over horizons up to six seconds.
- Trajectory-set representation: Dynamic trajectory sets approximate physically reachable states from the agent’s current state, whereas fixed sets return the same trajectories for every instance.The dynamic generator is a function of the current state; the fixed generator does not use it.
- Fixed trajectory sets: Fixed-set construction treats coverage as a greedy set-cover problem using maximum point-wise Euclidean distance between trajectories.The bagging algorithm repeatedly selects the candidate covering the most uncovered trajectories.
- Fixed trajectory sets: Fewer than 2,000 trajectories achieved under 2 meters of maximum point-wise ℓ2 distance for six-second trajectories in the experiments.The deterministic greedy procedure has a worst-case suboptimality factor of at most log (|K′|).
- Dynamic trajectory sets: Dynamic trajectory sets guarantee dynamical feasibility, while pruning with control profiles may not preserve 100% coverage of the sampled trajectory set.The pruning procedure applies a greedy covering strategy to control-input profiles rather than directly to trajectories.
- Dynamic trajectory sets: Dynamic sets are generated by forward-integrating diverse control sequences through a vehicle dynamics model, and can be sparser than fixed sets for equivalent coverage.Controls use constant lateral and longitudinal accelerations, with lateral acceleration normalized across speeds before conversion to steering angle.
4. Experiments
The experiments compare CoverNet and regression baselines using shared architectures, urban-driving data, trajectory metrics, and multiple fixed or dynamic trajectory-set configurations.
- Models: CoverNet models classify over fixed, dynamic, or hybrid trajectory sets, while MultiPath with dynamic anchors regresses residuals from speed-dependent anchors.Dynamic anchors are selected as a function of agent speed, then residuals are regressed from the chosen anchor.
- Implementation: The models use a ResNet-50 backbone combined with an agent state vector containing speed, acceleration, and yaw rate.The rasterized scene is encoded through global pooling before concatenation with the agent state representation.
- Implementation: CoverNet output dimensionality equals the number of trajectory modes, with hybrid nuScenes and internal-dataset splits of 92:682 and 524:500, respectively.These splits were selected to maximize coverage near ϵ ≈2 meters while minimizing the total number of categories.
- Datasets and setup: Experiments use 3-second and 6-second horizons, with sampling frequencies of 10 Hz for internal data and 2 Hz for nuScenes.The prediction horizon determines the number of predicted points through N = H × F.
- Metrics: Evaluation focuses on displacement error and hit rate over the most likely modes, including minADE_k, FDE, and HitRate_k,d.Hit rate records whether a prediction remains within distance d of the ground truth across the horizon.
5. Results
CoverNet outperforms prior methods across many metrics and datasets, although rankings vary by metric and performance can plateau on nuScenes with larger trajectory sets.
- CoverNet outperforms previous methods and baselines in 8 of 12 comparisons across six metrics and two datasets.Method rankings differ substantially depending on the metric.
- 33% HitRate5, 2m is achieved on nuScenes by the hybrid CoverNet trajectory set, versus 13% for the next-best MultiPath model.CoverNet’s dynamic grid extension slightly improves on MultiPath’s fixed grid, 13% versus 10%.
- 57% HitRate5, 2m is achieved by the 1,937-mode fixed CoverNet set on the internal dataset, followed closely by 55% for the hybrid set.The best prior method, dynamic MultiPath, reaches 30%.
- CoverNet outperforms state-of-the-art methods for minADEk with k ∈ {5, 10, 15} in every category.For internal data, hybrid CoverNet with 2,206 modes achieves minADE15 of 0.84.
- Performance improves with larger trajectory sets on the internal dataset but plateaus or decreases around 500–1,000 modes on nuScenes.The authors hypothesize that this difference is due to nuScenes’ relatively limited size.
- Figure 4 visualizes six-second predictions, including smooth trajectories and alternative left turns, while excluding straight trajectories because the vehicle slows before the curve.The scene’s apparent pedestrian collisions are avoided because pedestrians cross before the vehicle reaches the depicted pose.
6. Ablation studies
The ablations examine ground-truth matching and trajectory-set coverage, finding stable matching performance and more efficient coverage when dynamic trajectories are added.
- Performance is relatively consistent whether matching uses the maximum, average, or root-mean-square point-wise Euclidean error.The study selects the average point-wise ℓ2 norm to align better with related regression approaches.
- Dynamic vs fixed trajectory set coverage: Dynamic trajectories achieve the same coverage as fixed trajectories with fewer trajectories across different ε levels.Figure 3 compares trajectory counts required for 100% coverage using fixed and hybrid generators.
7. Conclusion
CoverNet frames urban trajectory prediction as classification over diverse trajectory sets, combining coverage, dynamic feasibility, and resistance to mode collapse.
- CoverNet classifies over trajectory sets to provide state-space coverage, eliminate dynamically infeasible trajectories, and avoid mode collapse.The trajectory sets remain manageable over realistic prediction horizons.
- Dynamically generating trajectory sets from the agent’s current state further improves performance.The method is evaluated against multiple state-of-the-art methods on public and internal real-world self-driving datasets.
A. Trajectory set visualization
Figure 6 visualizes how fixed, hybrid, and dynamic trajectory generators cover the trajectory space learned by CoverNet.
- Figure 6 shows fixed trajectory sets at different coverage levels and dynamic trajectory sets at one coverage level across varying speeds.
B. minADE1 over the prediction horizon
Figure 7 compares minADE1 over the prediction horizon for the best regression and CoverNet models on the internal dataset. The methods show similar performance over time for the most likely mode.
- All methods show similar performance over time for the most likely mode.
- Figure 7 plots minADE1 against prediction horizon for the best regression and CoverNet models on the internal dataset.
C. Greedy Approximation Algorithm
The paper uses a greedy approximation algorithm for solving equation (1), with a deterministic variant used in the work. A random variant can generate multiple bags and select the smallest cover set.
- The bagging algorithm is a greedy approximation for solving equation (1).
- The work uses a deterministic variant of the bagging algorithm.
- A random variant repeatedly forms bags through weighted choices based on uncovered trajectories and selects the cover set with the fewest elements.
- Figure 6 visualizes fixed and dynamic trajectory sets across coverage levels for a six-second prediction horizon.