Source-linked AI summary
Generalized Trajectory Scoring for End-to-end Multimodal Planning
Zhenxin Li, Wenhao Yao, Zi Wang, Xinglong Sun, Joshua Chen, Nadine Chang, Maying Shen, Zuxuan Wu, Shiyi Lan, Jose M. Alvarez
TL;DR
Existing trajectory scorers struggle to generalize between broad static vocabularies and precise dynamic proposals in end-to-end multimodal planning. GTRS unifies diffusion generation, dense-vocabulary dropout, sensor augmentation, and refinement, and its challenge-winning ensemble approaches a privileged ground-truth-perception planner despite synthetic sensor inputs.
Problem
Existing scorers specialize in large static vocabularies or small dynamic proposal sets, limiting generalization across diverse trajectory distributions.
Method
GTRS combines diffusion-based trajectory generation, dense-vocabulary training with dropout, and sensor augmentation with refinement for scoring static and dynamic candidates.
Results
GTRS-E reaches 49.4 EPDMS on Navhard, approaching PDM-Closed despite using challenging synthetic sensor input.
Takeaways & Limitations
GTRS supports trajectory scoring across static and dynamic distributions and robust performance across perceptual domains.
Takeaways & Limitations
Synthetic sensor data may contain distortion and blurring artifacts that impair sensor-based planners.
Abstract
from arXiv · showhide
End-to-end multi-modal planning is a promising paradigm in autonomous driving, enabling decision-making with diverse trajectory candidates. A key component is a robust trajectory scorer capable of selecting the optimal trajectory from these candidates. While recent trajectory scorers focus on scoring either large sets of static trajectories or small sets of dynamically generated ones, both approaches face significant limitations in generalization. Static vocabularies provide effective coarse discretization but struggle to make fine-grained adaptation, while dynamic proposals offer detailed precision but fail to capture broader trajectory distributions. To overcome these challenges, we propose GTRS (Generalized Trajectory Scoring), a unified framework for end-to-end multi-modal planning that combines coarse and fine-grained trajectory evaluation. GTRS consists of three complementary innovations: (1) a diffusion-based trajectory generator that produces diverse fine-grained proposals; (2) a vocabulary generalization technique that trains a scorer on super-dense trajectory sets with dropout regularization, enabling its robust inference on smaller subsets; and (3) a sensor augmentation strategy that enhances out-of-domain generalization while incorporating refinement training for critical trajectory discrimination. As the winning solution of the Navsim v2 Challenge, GTRS demonstrates superior performance even with sub-optimal sensor inputs, approaching privileged methods that rely on ground-truth perception. Code will be available at https://github.com/NVlabs/GTRS.
1. Introduction
End-to-end multimodal planning scores multiple trajectory candidates from raw sensor data, but existing static and dynamic scoring approaches generalize poorly across trajectory distributions. GTRS addresses this with complementary coarse- and fine-grained evaluation techniques, including diffusion proposals, vocabulary generalization, and sensor augmentation.
- Multimodal planners generate multiple trajectory candidates, supporting adaptable behavior across instructions, driving styles, and complex environments.
- The planner evaluates proposals from raw sensor data and selects the trajectory with the highest likelihood, without ground-truth perception.
- Static vocabularies offer coarse coverage but limited flexibility, whereas small dynamic proposal sets can fail on unseen trajectories.
- GTRS trains its scorer on both coarse and fine-grained trajectory distributions to improve generalization.
- Diffusion proposals provide fine-grained candidates, dense-vocabulary dropout promotes generalization, and sensor augmentation with refinement targets viewpoint shifts and subtle trajectory differences.
2. The Three Pillars of GTRS
GTRS combines diffusion-generated proposals, dense-vocabulary scoring, and sensor augmentation with refinement to evaluate diverse trajectories robustly. Its design spans proposal generation, vocabulary generalization, and fine-grained discrimination under varied sensor conditions.
- Diffusion-based Trajectory Generation: A diffusion policy generates N trajectory proposals conditioned on bird’s-eye-view features extracted from image data.The generator uses an image backbone, BEV encoder, and Diffusion Transformer.
- Trajectory Vocabulary Generalization: Diffusion proposals provide fine-grained candidates but do not capture the full breadth of possible driving scenarios, motivating vocabulary generalization.
- Trajectory Vocabulary Generalization: GTRS-Dense trains on VXL with 16,384 trajectories while inferring on VL with 8,192 trajectories to encourage generalizable representations.
- Trajectory Vocabulary Generalization: Vocabulary dropout randomly removes half of the dense vocabulary during training, and the resulting scorer handles both static vocabularies and dynamic proposals.
- Sensor Augmentation with Refinement: Sensor augmentation applies controlled horizontal view rotations with transformed ground truths, while refinement progressively improves scores for the top-k candidates.
- Sensor Augmentation with Refinement: An EMA teacher provides soft supervision for refinement, with clipping keeping refined targets within a reasonable range of ground-truth scores.
- Sensor Augmentation with Refinement: GTRS-Aug performs robustly in challenging out-of-domain settings without domain-specific adaptation.
3. Inference-time Integration
GTRS combines dynamically generated and fixed trajectory candidates only at inference time. This sequential design preserves broad scorer generalization while adding scene-specific fine-grained proposals without diffusion sampling during training.
- At inference, dynamic proposals Vdp are appended to the fixed vocabulary VL, and the combined set Vdp ∪ VL is tokenized and scored.The scorer evaluates the unified candidate set after proposal generation.
- Training on a diverse static vocabulary supports broad trajectory generalization, while diffusion supplies fine-grained, context-aware trajectories tailored to each scene.The generator contributes precision during deployment rather than during scorer training.
- Keeping diffusion sampling out of training avoids its computational overhead and potential instability while retaining dynamically generated trajectory precision at deployment.
4. Experiments
Experiments use Navhard, which combines difficult real-world scenarios with synthetic continuations, and evaluate trajectory scoring with EPDMS. GTRS variants generalize across trajectory distributions and perceptual domains, achieving their strongest result through ensembling.
- 4.1. Dataset and metrics: Navhard contains difficult real-world scenarios and synthetic continuations generated using 3DGS, providing the benchmark setting for evaluation.
- 4.1. Dataset and metrics: EPDMS aggregates multiple rule-based metrics through a two-stage pipeline covering real-world data in Stage 1 and synthetic data in Stage 2.
- 4.2. Implementation Details: Training uses Navtrain only; Navhard and other synthetic sensor data are excluded from training.Models are trained for 20 epochs by default, while the trajectory generator trains for 50 epochs.
- 4.3. Roadmap to Generalized Trajectory Scoring: EPDMS 43.4 is obtained with dropout-trained VXL, while combining Vdp with VXL adds +1.1 EPDMS over VXL.The results support vocabulary dropout and dynamic proposals as complementary inference components.
- 4.4. Main Results: 49.4 EPDMS is achieved by the challenge-winning GTRS-E ensemble, approaching privileged PDM-Closed despite using challenging synthetic sensor input.GTRS-E ensembles all six GTRS-Dense and GTRS-Aug models.