Source-linked AI summary
Parting with Misconceptions about Learning-based Vehicle Motion Planning
Daniel Dauner, Marcel Hallgarten, Andreas Geiger, Kashyap Chitta
TL;DR
The paper examines whether open-loop ego-forecasting and closed-loop driving provide aligned evidence for vehicle motion planning on nuPlan. It analyzes simple rule-based and centerline-conditioned planners, finding that their strengths differ across tasks, and combines them into PDM-Hybrid, which won the 2023 nuPlan competition.
Problem
Existing vehicle motion-planning systems struggle to satisfy precise short-term planning and long-horizon ego-forecasting requirements simultaneously.
Method
The paper evaluates state-of-the-art planners and develops an IDM-based closed-loop planner, a centerline-and-history-conditioned ego-forecaster, and their hybrid.
Results
PDM-Hybrid outperformed 24 competing approaches and won the 2023 nuPlan competition.
Takeaways & Limitations
Open-loop ego-forecasting and closed-loop planning should be addressed independently, with rule-based planners remaining promising for closed-loop driving.
Takeaways & Limitations
PDM does not execute lane changes, relies on HD maps and precise offboard perception, and was not specifically evaluated under distributional shifts beyond the held-out test set.
Abstract
from arXiv · showhide
The release of nuPlan marks a new era in vehicle motion planning research, offering the first large-scale real-world dataset and evaluation schemes requiring both precise short-term planning and long-horizon ego-forecasting. Existing systems struggle to simultaneously meet both requirements. Indeed, we find that these tasks are fundamentally misaligned and should be addressed independently. We further assess the current state of closed-loop planning in the field, revealing the limitations of learning-based methods in complex real-world scenarios and the value of simple rule-based priors such as centerline selection through lane graph search algorithms. More surprisingly, for the open-loop sub-task, we observe that the best results are achieved when using only this centerline as scene context (i.e., ignoring all information regarding the map and other agents). Combining these insights, we propose an extremely simple and efficient planner which outperforms an extensive set of competitors, winning the nuPlan planning challenge 2023.
1 Introduction
The nuPlan benchmark exposes a mismatch between open-loop ego-forecasting and closed-loop driving, while showing strong performance from simple rule-based and centerline-conditioned planners.
- Benchmark motivation: 1300 hours of real-world vehicle motion data in nuPlan enables large-scale benchmarking of vehicle motion planners.The benchmark includes an open-source dataset and simulator with official evaluation metrics.
- Evaluation misalignment: Open-loop ego-forecasting accuracy and closed-loop driving performance are negatively correlated.Learned planners excel at ego-forecasting but struggle with safe closed-loop plans, whereas rule-based planners show the opposite trend.
- Rule-based planning: A rule-based baseline from over twenty years ago surpasses all state-of-the-art learning-based methods on closed-loop evaluation metrics.This result challenges the claim that rule-based planning has difficulty generalizing.
- Centerline-conditioned planning: A centerline-only learned baseline achieves the new state of the art for open-loop evaluation without modeling other agents or complex scene representations.It uses the desired-route centerline to extrapolate the ego state and omits lane graphs, vectorized maps, rasterized maps, and tokenized objects.
- Combined contribution: PDM-Hybrid combines a lightweight IDM extension with a learned open-loop planner and outperforms 24 competing approaches in the nuPlan challenge 2023.The combined system integrates the paper’s closed-loop and open-loop findings into one simple baseline.
2 Related Work
Related work spans rule-based planners with explicit behavioral rules, hybrid systems that combine learned components with rule-based decisions, and ego-forecasting methods that directly predict future trajectories from observations.
- Rule-based planning: Rule-based planners use explicit rules to determine autonomous-vehicle behavior within a structured, interpretable decision-making framework.The Intelligent Driver Model is a seminal example designed to follow a leading vehicle while maintaining a safe distance.
- Hybrid planning: Hybrid planners combine rule-based decision-making with learned forecasts, affordance indicators, imitation learning, or safety filtering.These systems often forecast future environmental states to support informed and contingent driving decisions.
- Ego-forecasting: Ego-forecasting methods use observational data to directly determine the vehicle’s future trajectory rather than predictive planning.Approaches include end-to-end models using LiDAR or RGB inputs and modular methods using BEV grids or state vectors.
3 Ego-forecasting and Planning are Misaligned
On nuPlan, ego-forecasting accuracy and closed-loop driving performance are misaligned: improvements in one can degrade the other. The paper therefore combines a rule-based short-term planner with learned long-horizon corrections while retaining centerline-based structure.
- Misalignment: nuPlan evaluates ego-forecasting with open-loop accuracy and driving performance with closed-loop simulation, but improvements in one task can degrade the other.Open-loop metrics measure trajectory errors, whereas closed-loop metrics assess simulated driving outcomes such as progress and collisions.
- Rule-based baseline: IDM selects a centerline through lane-graph search and infers a longitudinal trajectory using manually selected acceleration, target-speed, safety-margin, and exponent parameters.The planner applies its policy using the ego’s longitudinal position, velocity, and distance to the leading vehicle.
- Ego-forecasting: PDM-Open predicts future waypoints from the extracted centerline and ego history, using a simple MLP designed for nuPlan’s high speeds and long forecasting horizons.Its inputs omit other agents and use a centerline sampled at one-meter resolution up to 120 meters.
- Misalignment: Table 1 shows a clear OLS–CLS trade-off: PDM-Open improves open-loop performance, while reducing IDM acceleration improves OLS but harms closed-loop performance.Adding the centerline contributes substantially to ego-forecasting, whereas history provides little improvement and lowers CLS.
- Closed-loop planning: PDM-Closed extends IDM with forecasting, trajectory proposals, simulation, scoring, and selection after graph-search centerline extraction.It forecasts agents with constant velocity, varies target speeds, simulates proposals with an LQR controller and kinematic bicycle model, and scores compliance, progress, comfort, and collision avoidance.
- Hybrid planning: PDM-Hybrid combines PDM-Closed’s short-term actions with learned long-term waypoint offsets, applying corrections beyond the first two seconds to preserve closed-loop behavior.The modular design allows individual components to be replaced with alternative options.
4 Experiments
Experiments show that simple rule-based and centerline-conditioned planners can outperform sophisticated learned baselines, while hybrid planning achieves strong overall benchmark performance. The results also expose trade-offs between open-loop accuracy, closed-loop driving, and runtime.
- Benchmark: The Val14 benchmark uses approximately 177k training scenarios and 1,118 evaluation scenarios across 14 leaderboard scenario types.Its validation split aligns with the online leaderboard evaluation, supporting its use as a proxy for the test set.
- Closed-loop results: IDM outperforms the best learned planner, while PDM-Closed improves closed-loop score from 76-77 to 92-93.The reported improvement results from the design ideas introduced for PDM-Closed.
- Open-loop results: PDM-Open achieves the highest OLS of 86 with a runtime of 7ms using only a centerline and ego state as input.It improves accurate long-horizon lane-following despite its minimal scene representation.
- Challenge: The preliminary graph version of PDM-Hybrid ranked first among 25 teams in the 2023 nuPlan challenge.The leaderboard averaged CLS-R, CLS-NR, and OLS; strong closed-loop performance outweighed slightly lower open-loop performance.
- Ablations: Removing PDM-Closed proposals reduces CLS-R but accelerates runtime, while excluding its forecasting causes a significant performance drop with nearly unchanged runtime.For PDM-Open, shorter centerlines and smaller networks degrade performance modestly, whereas coarser centerline sampling has negligible impact.
5 Discussion
The discussion argues that open-loop ego-forecasting and closed-loop planning are misaligned, despite both being used to assess motion planners. It recommends narrower use of long-horizon forecasting and identifies deployment and generalization boundaries for PDM.
- Discussion: Rule-based planning performs strongly on the closed-loop nuPlan task, while imitation-trained ego-forecasting methods perform poorly in closed-loop.The authors attribute this pattern to a trade-off between open-loop success and closed-loop performance.
- Discussion: PDM-Hybrid boosts open-loop performance without enhancing closed-loop driving, because it executes identical driving maneuvers.The authors therefore conclude that precise open-loop ego-forecasting is not a prerequisite for long-term planning goals.
- Evaluation scope: The authors recommend evaluating ego-forecasting over a short horizon such as 2 seconds rather than treating the 8-second OLS as the primary planning indicator.They retain possible uses for long-horizon OLS in alternate applications such as background-agent goals or computational resource allocation.
- Limitations: PDM does not execute lane changes and relies on HD maps and precise offboard perception that may be unavailable in real-world driving.The experiments also primarily use the nuPlan simulator and do not specifically test distribution shifts such as unseen towns or novel scenario types.
- Conclusion: PDM-Hybrid surpassed a comprehensive set of competitors and won the 2023 nuPlan competition.The conclusion presents the system as an IDM-based planner combined with a learned ego-forecasting component.