Source-linked AI summary
DenseTNT: End-to-end Trajectory Prediction from Dense Goal Sets
Junru Gu, Chen Sun, Hang Zhao
TL;DR
DenseTNT addresses multimodal and stochastic road-agent forecasting, where prior goal-based methods rely on sparse anchors and heuristic selection. It predicts dense goals anchor-free and end-to-end, using offline optimization to generate pseudo-labels for online training, and ranks first on Argoverse while winning the 2021 Waymo challenge.
Problem
Motion prediction is difficult because road-agent behavior is stochastic and multimodal, while existing goal-based methods depend on sparse anchors and heuristic goal selection.
Method
DenseTNT estimates dense goal probabilities without anchors, predicts a goal set end-to-end, and uses offline optimization to provide multi-future pseudo-labels for online training.
Results
DenseTNT achieves state-of-the-art performance, ranking 1st on the Argoverse motion forecasting benchmark and 1st in the 2021 Waymo Open Dataset Motion Prediction Challenge.
Takeaways & Limitations
DenseTNT combines dense goal modeling with online end-to-end prediction while achieving similar performance to its offline model and leading benchmark results.
Takeaways & Limitations
Sparse-anchor methods cannot generate multiple trajectory predictions around one anchor, capture fine-grained lane-segment information, or guarantee that greedy NMS finds the optimal multimodal goal set.
Abstract
from arXiv · showhide
Due to the stochasticity of human behaviors, predicting the future trajectories of road agents is challenging for autonomous driving. Recently, goal-based multi-trajectory prediction methods are proved to be effective, where they first score over-sampled goal candidates and then select a final set from them. However, these methods usually involve goal predictions based on sparse pre-defined anchors and heuristic goal selection algorithms. In this work, we propose an anchor-free and end-to-end trajectory prediction model, named DenseTNT, that directly outputs a set of trajectories from dense goal candidates. In addition, we introduce an offline optimization-based technique to provide multi-future pseudo-labels for our final online model. Experiments show that DenseTNT achieves state-of-the-art performance, ranking 1st on the Argoverse motion forecasting benchmark and being the 1st place winner of the 2021 Waymo Open Dataset Motion Prediction Challenge.
1. Introduction
DenseTNT addresses the difficulty of multimodal motion prediction by replacing sparse, heuristically selected goals with dense, anchor-free goal candidates and end-to-end goal-set prediction. It also uses offline optimization to create multi-future pseudo-labels, achieving leading benchmark performance.
- Motivation: Motion prediction is challenging because human behaviors are stochastic and multimodal, yet accurate forecasts support safer autonomous driving decisions.The passage gives vehicle cut-in prediction and braking as an example.
- Limitations of prior methods: Existing goal-based methods classify and regress predefined sparse anchors before applying heuristic selection such as NMS.These methods exploit the observation that trajectory uncertainty is concentrated largely in the endpoint.
- Limitations of prior methods: Sparse anchors cannot represent multiple goals around one anchor or fine-grained positional information along the same lane segment.Different positions may encode local cues such as distance to the nearest lane boundary.
- DenseTNT: DenseTNT estimates dense goal probabilities without heuristic anchors and generates the final goal set end-to-end.This removes rule-based post-processing from the goal-selection pipeline.
- Training: DenseTNT uses an offline optimization model to find goal sets that serve as multi-future pseudo-labels for training the online goal-set predictor.The online model predicts goal sets from the probability distribution, while the offline model uses optimization instead of the predictor.
- Results: DenseTNT achieves state-of-the-art performance, ranking 1st on the Argoverse motion forecasting benchmark and 1st in the 2021 Waymo challenge.The result is reported as the paper's headline benchmark outcome.
2. Related Work
Prior work models multimodal futures with latent-variable sampling, raster or vector map encoders, and goal-based pipelines. DenseTNT extends this line with dense spatial goal modeling and anchor-free end-to-end prediction.
- Multimodal prediction: Future trajectories are uncertain because agent intents, behaviors, and interactions are unknown or complicated.Latent variables are widely used to model this multimodality.
- Map encoding: Rasterized encoders represent maps and agents as images, while vectorized encoders represent lanes and agents as sparse elements processed with graph-based models.Examples include CNN-based raster methods and graph-based methods such as VectorNet and LaneGCN.
- DenseTNT: Unlike methods focused on lane centerlines or boundaries, DenseTNT models dense spatial locations on roads.This supplies denser spatial information for goal estimation.
- Goal-based prediction: Goal-based methods predict endpoints before completing trajectories, often sampling sparse anchors and using NMS to select final trajectories.TNT and LaneRCNN are described as representative anchor-based pipelines.
- DenseTNT: DenseTNT is an anchor-free goal-based model learned end-to-end, while concurrent HOME similarly uses dense probability estimation but greedy goal sampling.The comparison identifies both methodological similarity and a distinction in goal selection.
3. Method
DenseTNT replaces sparse anchor-based goal selection with dense goal probabilities and an end-to-end goal set predictor. An offline optimization procedure supplies multi-future pseudo-labels for training the online predictor.
- Context encoding: DenseTNT encodes map and agent context with sparse vectorized features before estimating goals on the road.VectorNet’s subgraph and global graph modules encode entities and their interactions into map-element features.
- Dense goal probability estimation: Dense goal estimation represents road locations as dense candidates, enabling anchor-free probability prediction and reducing reliance on sparse anchors.A lane-scoring module filters candidates to likely lanes before dense goal encoding and probability estimation.
- Goal set prediction: The goal set predictor replaces heuristic NMS by directly selecting a final set of goals from the predicted heatmap in an end-to-end manner.NMS may require different thresholds for different heatmaps, producing suboptimal selections.
- Two-stage training: DenseTNT trains the goal set predictor with pseudo-labels generated offline because each trajectory-prediction sample provides only one observed future.The offline model uses optimization to provide multi-future supervision for the online goal set predictor.
- Offline optimization: The offline objective minimizes expected displacement error over the heatmap distribution, using hill climbing to search for a low-error goal set.The search begins from a randomly sampled goal set and iteratively perturbs candidate goals while evaluating the expected error.
4. Experiments
DenseTNT is evaluated on Argoverse and Waymo, using standard forecasting metrics, benchmark comparisons, qualitative predictions, and ablations of its components and optimization settings.
- Datasets: Argoverse contains 333K driving sequences, while Waymo provides over 100,000 20-second scenes spanning vehicles, pedestrians, and cyclists.Argoverse predicts three seconds from two seconds of history; Waymo predicts eight seconds from one second of history.
- Metrics: Evaluation follows Argoverse metrics minADE, minFDE, and miss rate.minADE and minFDE select the best among K predicted trajectories, while miss rate counts cases where all final predictions exceed 2.0 meters from ground truth.
- Benchmark results: On Argoverse, the online goal-set model achieves comparable results to the offline optimization model and superior performance on the official MR ranking metric.DenseTNT also achieves state-of-the-art minFDE when minFDE is used as the optimization objective.
- Benchmark results: DenseTNT ranks first on the Argoverse motion forecasting benchmark and first in the 2021 Waymo Open Dataset Motion Prediction Challenge.The Waymo leaderboard reports mAP as its official ranking metric.
- Ablation study: Ablations show that dense probability estimation provides finer local information than sparse estimation, while the online model performs almost the same as the offline model.The end-to-end method outperforms variety loss by a large margin.
- Ablation study: Higher goal density improves performance until saturation, and optimization performance rises drastically before t = 100ms before remaining almost unchanged after t = 200ms.These findings are reported for Argoverse validation experiments under different sampling densities and optimization times.
5. Conclusion
DenseTNT is an anchor-free, end-to-end model that outputs trajectories from dense goal candidates and uses offline optimization to provide multi-future pseudo-labels. It achieves state-of-the-art performance on Argoverse and Waymo benchmarks.
- DenseTNT directly outputs a set of trajectories from dense goal candidates without predefined anchors.
- An optimization-based offline model provides multi-future pseudo-labels for training the online model.
- DenseTNT has similar performance to the offline model, supporting the goal set predictor and training paradigm.
- DenseTNT ranks 1st on the Argoverse motion forecasting benchmark and wins 1st place in the 2021 Waymo Open Dataset Motion Prediction Challenge.
A. Offline Optimization
The offline model combines context encoding with an optimization algorithm to strengthen DenseTNT training. Its optimization objectives are evaluated across different combinations using multiple trajectory-prediction metrics.
- The offline model consists of a context encoding module and an optimization algorithm.
- The optimization algorithm’s effectiveness is tested under different combinations of optimization objectives.
- The evaluation uses multiple metrics to assess multi-trajectory prediction comprehensively.
B. Implementation Details
Implementation normalizes map and agent inputs into vector sequences, searches goal sets by expected error, and encodes sufficiently probable heatmap goals for goal-set prediction.
- The map and agents are normalized around the target vehicle and represented as sequences of vectors with geometric and attribute information.Lane vectors include lane indices, while agent vectors include timestamps of their observations.
- The optimization algorithm searches for a goal set minimizing expected error and evaluates hundreds of candidates in 100ms using eight CPUs in parallel.Different initializations are used, and the best result is selected.
- Expected error calculation considers heatmap goals whose probabilities satisfy h(c_i) ≥ 10^-3.
- Each 1m × 1m heatmap goal is divided into nine subgoals to obtain a more precise expected error.
- The goal set predictor encodes heatmap goals with h(c_i) ≥ 10^-5 using normalized coordinates and log probabilities.A 2-layer MLP processes the encoded heatmap-goal features.
C. Qualitative Results
Qualitative results compare DenseTNT with TNT and show its predictions across diverse traffic scenarios. The visualizations emphasize dense goal distributions, selected goal sets, trajectories, and ground truth, while diverse final-position probabilities challenge NMS.
- DenseTNT qualitative results include cases where the final-position probability distribution is highly diverse.The passage states that NMS is difficult to handle well in such cases.
- Table 6 reports optimization-algorithm performance under different combinations of optimization objectives.
- Figure 6 compares TNT in the upper row with DenseTNT online in the lower row on the Argoverse validation set.Anchors and heatmaps are red, predicted trajectories are orange, and ground truth trajectories are green.
- Figure 7 presents DenseTNT online across diverse traffic scenarios on the Argoverse validation set.Dense heatmaps are red, predicted goal sets and trajectories are orange, and ground truth trajectories are green.