Source-linked AI summary
Hydra-MDP: End-to-end Multimodal Planning with Multi-target Hydra-Distillation
Zhenxin Li, Kailin Li, Shihao Wang, Shiyi Lan, Zhiding Yu, Yishen Ji, Zhiqi Li, Ziyue Zhu, Jan Kautz, Zuxuan Wu, Yu-Gang Jiang, Jose M. Alvarez
TL;DR
End-to-end autonomous-driving planners face limitations in open-loop imitation-learning evaluation, imperfect perception for rule-based planning, and non-differentiable post-processing. Hydra-MDP uses multi-target teacher-student distillation with human and rule-based teachers to learn diverse metric-specific trajectories end to end, achieving state-of-the-art performance on Navsim simulation-based metrics.
Problem
Open-loop imitation-learning evaluation has dysfunctional metrics and implicit biases, while rule-based planning relies on imperfect predicted perceptions.
Method
Hydra-MDP uses teacher-student distillation from human and rule-based planners, with a multi-target paradigm that predicts diverse metric-specific trajectories from sensor observations.
Results
Hydra-MDP achieves state-of-the-art performance under the simulation-based evaluation metrics on Navsim.
Takeaways & Limitations
Multi-target hydra-distillation aligns end-to-end planning with simulation-based metrics while avoiding reliance on non-differentiable post-processing.
Takeaways & Limitations
DDC is neglected for distillation and subsequent results because of an implementation problem.
Abstract
from arXiv · showhide
We propose Hydra-MDP, a novel paradigm employing multiple teachers in a teacher-student model. This approach uses knowledge distillation from both human and rule-based teachers to train the student model, which features a multi-head decoder to learn diverse trajectory candidates tailored to various evaluation metrics. With the knowledge of rule-based teachers, Hydra-MDP learns how the environment influences the planning in an end-to-end manner instead of resorting to non-differentiable post-processing. This method achieves the $1^{st}$ place in the Navsim challenge, demonstrating significant improvements in generalization across diverse driving environments and conditions. More details by visiting \url{https://github.com/NVlabs/Hydra-MDP}.
1. Introduction
End-to-end autonomous driving must address safety-relevant closed-loop criteria, but existing imitation-learning and post-processing approaches have important limitations. Hydra-MDP addresses this with multi-target, multimodal teacher-student distillation from human and rule-based planners.
- Open-loop imitation-learning evaluation can suffer from dysfunctional metrics and implicit biases, failing to guarantee safety, efficiency, comfort, and traffic-rule compliance.
- End-to-end planning is multi-target because it must satisfy diverse open-loop and closed-loop metrics, and multimodal because each metric may admit multiple optimal solutions.
- Rule-based planners degrade with imperfect predicted perceptions, while post-processing closed-loop evaluation is not fully streamlined and may discard information.
- Hydra-MDP trains a student to learn diverse trajectory candidates for different metrics through knowledge distillation from human and rule-based teachers.
- The extendable distillation architecture supports integrating additional teachers, while environmental-observation training prepares the student for testing without ground-truth perception.
- Hydra-MDP achieves state-of-the-art performance under simulation-based Navsim evaluation metrics.
2. Solution
Hydra-MDP replaces single-target planning and non-differentiable trajectory selection with end-to-end multi-target learning through teacher-student distillation. Its architecture combines multimodal perception, trajectory vocabularies, human imitation, and rule-based simulation metrics.
- Preliminaries: Single-modal planning directly regresses trajectories from sensor observations, while multimodal single-target planning selects candidates through a non-differentiable cost function using predicted perception.The selected trajectory is the candidate with the lowest cost, but the process relies on imperfect perception.
- Preliminaries: Hydra-MDP jointly predicts multiple trajectory costs with a neural network and distills teacher knowledge while the student uses only sensor observations.Teachers access ground-truth perception, whereas the student learns from environmental observations.
- Preliminaries: The framework scales end-to-end by adding cost functions or imitation similarity instead of relying on non-differentiable post-processing.This design supports multi-target learning within the planning network.
- Overall Framework: The system combines a perception network with a trajectory decoder that uses image and LiDAR features, environmental tokens, and a fixed trajectory vocabulary.The vocabulary is formed by K-means clustering 700K trajectories sampled from nuPlan, with each trajectory covering 40 timestamps over a 4-second horizon at 10Hz.
- Overall Framework: Human imitation supervision uses L2 distances between log-replay trajectories and vocabulary candidates to produce a probability distribution favoring trajectories close to human driving behavior.Softmax converts the distance-based scores into imitation targets.
- Multi-target Hydra-Distillation: Multi-target Hydra-Distillation addresses closed-loop failures by simulating vocabulary trajectories offline and distilling per-metric simulation scores into Hydra prediction heads.The learned sub-scores act as costs associated with violations of particular traffic rules, and confidence weights combine them for trajectory selection.
- Multi-target Hydra-Distillation: The framework also supports confidence-weighted cost assembly and model ensembling through Mixture of Encoders and Sub-score Ensembling.Sub-score Ensembling combines weighted sub-scores from independent models for trajectory selection.
3. Experiments
Experiments evaluate Hydra-MDP on Navsim using the PDM score and its component metrics, while testing vocabulary size, learning targets, inference weighting, and backbone scaling. Results support multi-target distillation and show that larger backbones can improve the model's performance.
- Dataset: Navtest contains 136 scenarios, while Navtrain contains 1192 scenarios for training and validation.The dataset focuses on scenarios involving changes in driving intention and provides annotated 2D high-definition maps and 3D object bounding boxes.
- Metrics: PDMscore combines NC, DAC, DDC, TTC, C, and EP, covering collisions, drivable-area compliance, collision timing, comfort, and ego progress.NC, DAC, TTC, C, and EP are defined in the supplied metrics description; DDC is omitted from distillation and subsequent results.
- Ablations: Hydra-MDP uses a larger planning vocabulary V8192, weighted confidence, and multi-target learning to improve performance across the evaluated methods and metrics.Distilling the overall PDM score alone suffers from irregular score distribution, whereas distilling EP improves the corresponding metric in the final Hydra-MDP-V8192-W-EP version.
- Ablations: Non-differentiable post-processing yields fewer performance gains than Hydra-MDP in the reported experiments.The comparison is made in the context of the Navtest results summarized in Table 1.
- Scaling: Three best-performing Hydra-MDP versions use ViT-L and V2-99 image backbones, whose ensembled sub-scores are used for final inference.The scaling experiment evaluates larger image backbones on the Navtest split.