Source-linked AI summary
NuPlan: A closed-loop ML-based planning benchmark for autonomous vehicles
Holger Caesar, Juraj Kabzan, Kok Seang Tan, Whye Kit Fong, Eric Wolff, Alex Lang, Luke Fletcher, Oscar Beijbom, Sammy Omari
TL;DR
Existing autonomous-driving benchmarks emphasize short-term prediction and lack standardized data, metrics, and closed-loop evaluation for long-term planning. This work introduces a real-world planning benchmark combining a large multi-city dataset, closed-loop evaluation with reactive agents, and planning-specific metrics; its principal contribution is a common benchmark framework for ML-based planning.
Problem
Published benchmarks focus on short-term prediction rather than long-term planning, while suitable datasets, standardized metrics, and closed-loop evaluation remain limited.
Method
The benchmark combines a large real-world driving dataset, a closed-loop evaluation framework with reactive agents, and general and scenario-specific planning metrics.
Results
The work proposes the first public benchmark for real-world data with a closed-loop planner evaluation protocol.
Takeaways & Limitations
The benchmark provides a common framework intended to support progress in ML-based planning for autonomous vehicles.
Takeaways & Limitations
Simulations introduce systematic biases, and the proposed metrics are an initial, non-exhaustive list.
Abstract
from arXiv · showhide
In this work, we propose the world's first closed-loop ML-based planning benchmark for autonomous driving. While there is a growing body of ML-based motion planners, the lack of established datasets and metrics has limited the progress in this area. Existing benchmarks for autonomous vehicle motion prediction have focused on short-term motion forecasting, rather than long-term planning. This has led previous works to use open-loop evaluation with L2-based metrics, which are not suitable for fairly evaluating long-term planning. Our benchmark overcomes these limitations by introducing a large-scale driving dataset, lightweight closed-loop simulator, and motion-planning-specific metrics. We provide a high-quality dataset with 1500h of human driving data from 4 cities across the US and Asia with widely varying traffic patterns (Boston, Pittsburgh, Las Vegas and Singapore). We will provide a closed-loop simulation framework with reactive agents and provide a large set of both general and scenario-specific planning metrics. We plan to release the dataset at NeurIPS 2021 and organize benchmark challenges starting in early 2022.
1. Introduction
Existing benchmarks largely evaluate short-term prediction rather than goal-based, interactive planning, motivating a dedicated planning benchmark with closed-loop evaluation and planning-specific metrics.
- Motivation: Existing real-world benchmarks target short-term motion forecasting rather than ego-vehicle planning, lacking high-level goals and closed-loop interaction.Prediction concerns other agents, whereas planning concerns the ego vehicle.
- Motivation: L2 displacement metrics can penalize valid but unobserved choices in multimodal driving situations.For example, merging immediately or later may both be valid, although only one appears in the observed trajectory.
- Contributions: The benchmark contributes a large public real-world dataset with high-quality autolabeled tracks from 4 cities.The dataset contribution is paired with planning metrics covering traffic rules, human similarity, dynamics, goals, and scenarios.
- Contributions: It provides the first public benchmark for real-world data with a closed-loop planner evaluation protocol.The benchmark is designed specifically to address shortcomings in existing planning evaluation.
2. Related Work
Prior work spans prediction datasets, planning benchmarks, simulators, and ML planners, but lacks a common real-world framework combining published data, standardized metrics, and closed-loop evaluation.
- Datasets: Existing motion datasets emphasize prediction rather than planning, while CommonRoad provides planning scenarios without sensor data at a scale unsuitable for modern deep learning.CommonRoad contains 5700 scenarios but does not support training modern deep learning methods according to the passage.
- Positioning: The benchmark therefore combines real-world data with planning metrics and closed-loop evaluation to overcome the prediction focus of prior datasets.This distinction separates the benchmark from existing prediction-oriented datasets.
- Simulation: CARLA supports flexible sensor and environmental configurations, but its challenge does not use real-world data and provides less detailed planning metrics.The comparison concerns the CARLA Autonomous Driving Challenge and the proposed benchmark.
- Simulation: Simulators have fundamental limits because they introduce systematic biases, and realistic sensor emulation remains an open requirement.The passage specifically cites photorealistic camera-image generation as an area requiring further work.
- ML-based planning: ML-based planning lacks a common input/output space, dataset, and metric set across the literature.Existing works report inconsistent measures such as infractions, route completion, off-road time, and destination success.
- ML-based planning: The proposed benchmark addresses this gap with an ML-based planning dataset and metrics for closed-loop evaluation.The authors position this as a response to limited progress caused by missing published datasets and standardized metrics.
3. Dataset
The dataset contains 1500 hours of multimodal driving data across four geographically diverse cities, with maps, sensor inputs, autolabeled trajectories, and scenario annotations.
- Dataset: 1500 hours of data span Las Vegas, Boston, Pittsburgh, and Singapore, whose traffic patterns present distinct driving challenges.Examples include casino pickup areas, double parking, custom left-turn precedence, and left-hand traffic.
- Dataset: Each city includes semantic maps and an API for efficient map queries.These map resources accompany the city-diverse driving data.
- Dataset: The dataset provides lidar point clouds, camera images, localization information, and steering inputs.These are the listed sensor and vehicle-input modalities.
- Autolabeling: An offline perception system produces high-accuracy autolabeled trajectories without online real-time constraints.The system combines PointPillars, CenterPoint, modified multi-view fusion, and non-causal tracking for near-human labeling performance.
- Scenarios: Automatically tagged intervals support scenario-based metrics for merges, turns, vulnerable-road-user interactions, construction zones, and other complex situations.The listed scenarios also include lane changes, double parked vehicles, stop-controlled intersections, close proximity, and high acceleration.
4. Benchmarks
The benchmark evaluates ML-based planners through open-loop and closed-loop tasks, using common and scenario-specific metrics that cover safety, human similarity, dynamics, progress, and challenging maneuvers. Its metric set is an initial proposal intended for community refinement.
- Evaluation protocol: Containerized planner submissions enable portable closed-loop evaluation on a secret test set.Planners operate on autolabeled trajectories or directly on raw sensor data for end-to-end open-loop approaches.
- Tasks: Benchmark challenges increase in difficulty from human-driver imitation in open loop to closed-loop planning evaluation.The open-loop task scores trajectories without using them to control the vehicle, whereas closed-loop tasks use planned trajectories as controller references.
- Metrics: Common metrics measure traffic-rule compliance, human-driving similarity, vehicle dynamics, and goal achievement.They include collision and off-road rates, time-gap and collision measures, trajectory errors relative to humans, comfort and feasibility criteria, and route progress toward a waypoint.
- Metrics: Scenario-based metrics target lane changes, pedestrian and cyclist interactions, crosswalks, and unprotected turns.These metrics assess safety gaps and collision times, passing relative velocity, and agreement between planner and human right-of-way decisions.
- Community feedback: The proposed metric list is not exhaustive, and the community is invited to add scenarios and agree on the main challenge metric.Possible aggregate metrics include weighted sums, thresholded violation sums, or metric hierarchies.
5. Conclusion
The work proposes an ML-based planning benchmark for autonomous vehicles focused on goal-based planning, planning metrics, and closed-loop evaluation. The authors hope a common benchmark will support progress in ML-based planning.
- The benchmark focuses on goal-based planning, planning metrics, and closed-loop evaluation rather than existing forecasting benchmarks.
- The authors hope the common benchmark will pave a path toward progress in ML-based planning.