Source-linked AI summary
End-to-end Interpretable Neural Motion Planner
Wenyuan Zeng, Wenjie Luo, Simon Suo, Abbas Sadat, Bin Yang, Sergio Casas, Raquel Urtasun
TL;DR
Autonomous driving needs reliable planning in complex urban settings, while existing end-to-end and modular approaches have limitations in interpretability, data demands, or joint optimization. The paper proposes an interpretable neural planner that learns representations and a space-time cost volume from LiDAR and HD maps, samples feasible trajectories, and selects the minimum-cost one; experiments on real-world data from several North American cities show safer planning than baselines.
Problem
Existing end-to-end driving approaches suffer from compounding error, massive data requirements, and difficult interpretability, while modular stacks may yield sub-optimal overall performance when subtasks are optimized separately.
Method
The planner jointly processes LiDAR point clouds and an HD map into 3D detections, future trajectories, and a space-time cost volume, then samples feasible trajectories and selects the minimum-cost proposal.
Results
The model demonstrates better detection and motion forecasting performance than recent task-specific neural architectures and safer planning than baselines on real-world driving data from several North American cities.
Takeaways & Limitations
A learned cost volume provides interpretable intermediate representations and naturally captures uncertainty and multimodality in possible SDV trajectories.
Takeaways & Limitations
Under physical-feasibility constraints, the trajectory-minimization optimization is NP hard, so inference relies on sampling a low-cost trajectory.
Abstract
from arXiv · showhide
In this paper, we propose a neural motion planner (NMP) for learning to drive autonomously in complex urban scenarios that include traffic-light handling, yielding, and interactions with multiple road-users. Towards this goal, we design a holistic model that takes as input raw LIDAR data and a HD map and produces interpretable intermediate representations in the form of 3D detections and their future trajectories, as well as a cost volume defining the goodness of each position that the self-driving car can take within the planning horizon. We then sample a set of diverse physically possible trajectories and choose the one with the minimum learned cost. Importantly, our cost volume is able to naturally capture multi-modality. We demonstrate the effectiveness of our approach in real-world driving data captured in several cities in North America. Our experiments show that the learned cost volume can generate safer planning than all the baselines.
1. Introduction
The paper bridges end-to-end driving and traditional engineering stacks with an end-to-end learnable, interpretable motion planner. It represents perception, future motion, and planning costs jointly before selecting a minimum-cost trajectory.
- Motivation: End-to-end driving is simple to build but suffers from compounding error, requires massive data, and is difficult to interpret or constrain with scene knowledge.Traditional stacks divide perception, prediction, planning, and control into subtasks, but isolated objectives can produce sub-optimal overall performance.
- Contribution: The paper proposes bridging end-to-end and traditional-stack frameworks with an end-to-end learnable and interpretable motion planner.The model is designed to connect learned driving with structured intermediate representations and planning.
- Contribution: The planner takes LiDAR point clouds and an HD map, producing 3D detections, future trajectories, and a space-time cost volume over possible SDV locations.The cost volume represents the goodness of locations within the planning horizon.
- Contribution: It samples diverse feasible trajectories and selects the trajectory with the minimum learned cost.The architecture combines intermediate predictions with cost-based trajectory selection.
- Results: The model demonstrates better detection and forecasting performance than recent task-specific neural architectures and safer planning than baselines on North American real-world driving data.The evaluation covers data captured in several cities and reports safer planning relative to the baselines.
2. Related Work
Prior approaches include imitation, reinforcement, optimization-based, uncertainty-aware, and holistic planners, each with stated limitations in data, scalability, interpretability, or manual design. This work extends holistic modeling with a jointly trained learned cost map that handles multimodality and uncertainty.
- Imitation Learning: Imitation-learning approaches directly map sensor data or commands to actions, but suffer from compounding error, require substantial data, and generalize poorly to situations such as drifting out of lane.Some variants add high-level commands or road segmentation as intermediate representations.
- RL & IRL: Reinforcement- and inverse-reinforcement-learning methods address sequential decisions or learn reward functions, but reported testing is limited and their scalability to realistic settings is unclear.The cited approaches also do not produce interpretable representations, which are desirable in safety-critical applications.
- Optimization Based Planners: Optimization-based planners usually minimize manually engineered costs built on perception and prediction outputs, often simplifying the search space or objective.Examples include quadratic objectives, separate lateral and longitudinal planning, speed-path representations, graph search, dynamic programming, and continuous optimization.
- Planning under uncertainty: Uncertainty-aware planners model uncertainty in other actors’ intentions, future actions, or collision probabilities, whereas this planner learns a non-parametric cost function to handle uncertainty.The comparison is stated as a distinction between explicit uncertainty modeling and cost-based handling.
- Holistic Models: The proposed holistic model predicts a planning cost map from raw sensors and dynamic map data, is jointly trained for the end task, and handles multimodality without costly parameter tuning.Compared with direct steering regression, it provides interpretability; compared with traditional planners, it learns representations jointly with planning.
3. Deep Structured Interpretable Planner
The planner jointly learns interpretable perception, forecasting, and planning from LiDAR and HD maps. It represents future location goodness with a cost volume, then selects the lowest-cost physically feasible trajectory while training perception and planning together.
- Model overview: The model takes LiDAR point clouds and an HD map, producing 3D detections, future motion forecasts, and a space-time cost volume.The cost volume assigns goodness to possible SDV locations over the planning horizon.
- Learning objective: End-to-end training combines a planning loss with perception losses for accurate 3D detections and motion forecasts.The planning loss treats the human trajectory as positive and sampled trajectories as negatives, penalizing traffic-rule violations more severely.
- Cost volume: The cost volume contains a two-dimensional cost map for each future timestep and is produced at BEV resolution by the cost-volume head.A final convolution layer with T filters generates one cost map per planning timestep.
- Trajectory selection: Because direct cost minimization is NP hard under physical constraints, the planner samples physically possible trajectories and chooses the one with minimum learned cost.Trajectories are represented through spatial paths and velocity profiles rather than arbitrary Cartesian point sequences.
- Trajectory parameterization: The trajectory sampler enforces vehicle dynamics with a bicycle model and represents paths using Clothoid curves.The bicycle model links path curvature to steering angle, while Clothoid curvature changes linearly along the path.
4. Experiments
Experiments evaluate the planner on large-scale North American driving data using planning, interpretability, and ablation analyses. The model achieves safer planning, strong detection and forecasting, and time-dependent cost maps capturing several driving behaviors.
- Evaluation Setup: The dataset contains 6,500 scenarios and approximately 1.4 million frames collected across multiple North American cities.The training, validation, and test sets contain 5,000, 500, and 1,000 scenarios, respectively.
- Planning Results: Planning evaluation measures trajectory distance, future potential collision rate, and traffic-rule violations against multiple baselines.The baselines include ego-motion forecasting, imitation learning, and manually designed cost planning.
- Planning Results: Our approach has lower future collision rates at all timesteps and combines better regression numbers with better collision rates than the manual-cost baseline and ACC.It also achieves smaller lane-violation numbers than other baselines, while ACC has approximately zero lane violations by design.
- Interpretability: The model achieves the best detection results at the 0.7 IoU threshold, identified as the metric of choice for self-driving.Qualitative detection results are also shown in Figure 3.
- Interpretability: Motion forecasting outperforms the LiDAR-based baselines on every reported metric and at every timestep.The metrics include L1 and L2 distances, plus errors along and perpendicular to the ground-truth trajectory.
- Interpretability: The learned cost map is time-dependent and illustrates multi-modality, lane-following in heavy traffic, and collision avoidance across example scenarios.Figure 3 overlays lower-cost regions for different future timesteps using different colors, alongside detections, predictions, planned trajectories, and ground truth.
- Ablation Study: Ablations show that longer input sequences improve results, planning-only training performs worse, and removing the traffic-rule penalty worsens planning.The best model has similar detection and forecasting performance to the detection-and-forecasting-only model; the penalty provides prior knowledge for collision avoidance.
5. Conclusion
The paper proposes a neural motion planner that learns safe, traffic-rule-following driving in complex real-world scenarios. It combines interpretable predictions with a learned cost map and selects the minimum-cost physically possible trajectory.
- The holistic planner takes LiDAR data and an HD map, producing 3D detections, future trajectories, and a cost map over positions within the planning horizon.It samples physically possible trajectories and chooses the one with minimum learned cost.
- The approach demonstrates effective and accurate driving in complex real-world scenarios across several North American cities.