PaperlayerView exact revision on arXiv

Source-linked AI summary

Towards learning-based planning:The nuPlan benchmark for real-world autonomous driving

Napat Karnchanachari, Dimitris Geromichalos, Kok Seang Tan, Nanxiang Li, Christopher Eriksen, Shakiba Yaghoubi, Noushin Mehdipour, Gianmarco Bernasconi, Whye Kit Fong, Yiluan Guo, Holger Caesar

arXiv:2403.04133v1cs.CVcs.RO

TL;DR

Learning-based planning still faces generalization challenges across diverse driving situations, while its safety guarantees remain limited. nuPlan combines a large real-world dataset, auto-labeling, and closed-loop simulation to evaluate planners; its challenge results favor rule-based planners over purely ML-based ones, with hybrid planners showing the most promise in difficult scenarios.

  • Problem

    Learning-based planning has not matched rule-based methods and faces generalization challenges from limited examples and long-tail driving scenarios, as well as a lack of formal safety guarantees.

    Source: §I · p. 1

  • Method

    nuPlan provides a real-world driving dataset with auto-labeled tracks and traffic lights, scenario mining, and a closed-loop simulation and evaluation framework for comparing planners.

    Source: §I · p. 1, A. Agents · p. 4

  • Results

    The first public nuPlan challenge found that rule-based planners outperform purely ML-based planners, while hybrid planners with learned components show the most promise in difficult scenarios.

    Source: §VI · p. 6

  • Takeaways & Limitations

    nuPlan offers a real-world dataset and closed-loop evaluation framework for investigating planner performance across challenging driving scenarios.

    Source: §I · p. 1, §VI · p. 6

Abstract

from arXiv · show

Machine Learning (ML) has replaced traditional handcrafted methods for perception and prediction in autonomous vehicles. Yet for the equally important planning task, the adoption of ML-based techniques is slow. We present nuPlan, the world's first real-world autonomous driving dataset, and benchmark. The benchmark is designed to test the ability of ML-based planners to handle diverse driving situations and to make safe and efficient decisions. To that end, we introduce a new large-scale dataset that consists of 1282 hours of diverse driving scenarios from 4 cities (Las Vegas, Boston, Pittsburgh, and Singapore) and includes high-quality auto-labeled object tracks and traffic light data. We exhaustively mine and taxonomize common and rare driving scenarios which are used during evaluation to get fine-grained insights into the performance and characteristics of a planner. Beyond the dataset, we provide a simulation and evaluation framework that enables a planner's actions to be simulated in closed-loop to account for interactions with other traffic participants. We present a detailed analysis of numerous baselines and investigate gaps between ML-based and traditional methods. Find the nuPlan dataset and code at nuplan.org.

I. INTRODUCTION

nuPlan addresses the challenges facing learning-based planning with a real-world dataset and simulation framework for comparing traditional, learning-based, and hybrid planners. The paper situates this testbed among datasets and simulators with differing planning, data, and simulation capabilities.

I. INTRODUCTION

  • Learning-based planning faces generalization challenges from limited examples and long-tail scenarios, alongside a lack of formal safety guarantees.The introduction presents nuPlan as a testbed for comparing traditional, learning-based, and hybrid planners.Source: §I · p. 1

II. RELATED WORK

  • The related-work section frames nuPlan against datasets and simulators for planning, then introduces classical, learning-based, and hybrid planner categories.It notes that most datasets in its comparison focus on prediction rather than planning.Source: §II · p. 1, A. Datasets · p. 1

A. Datasets

  • nuPlan is the first dataset described as offering both scenario tags and a closed-loop simulation framework for evaluating planning in specific scenarios.The comparison also identifies offline perception and traffic-light-status availability as important dataset capabilities.Source: A. Datasets · p. 1

B. Simulation

  • nuPlan combines a large real-world driving dataset with an open-source closed-loop simulator, addressing limitations of proprietary and graphical simulators and scenario-based alternatives.The related-work discussion notes that CommonRoad lacks a real-world dataset basis, while graphical simulators lack realism in maps and agent behavior.Source: B. Simulation · p. 2

C. Planning

The paper reviews classical, learning-based, and hybrid planning approaches, then describes nuPlan’s real-world data collection across four cities. Its released data include a subset of sensor recordings and detailed semantic maps.

C. Planning

  • Classical planners optimize handcrafted costs, learning-based approaches include imitation and reinforcement learning, and hybrid methods combine learning with classical planning.The section also discusses multi-task learning and unified end-to-end system designs for perception, prediction, and planning.Source: C. Planning · p. 2

A. Data collection

  • nuPlan collects 1,282 hours of real-world driving across Boston, Pittsburgh, Las Vegas, and Singapore, including city-specific situations such as double parking and left-hand traffic.Heavy-rain and night data are excluded because they would affect perception quality.Source: A. Data collection · pp. 2–3
  • Operators manually drove Chrysler Pacifica vehicles in a natural, safe style along predefined routes, unlike datasets mixing manual and automated driving.The authors state that such mixing may lead planners to imitate less desirable driving behavior.Source: A. Data collection · p. 3
  • The full sensor dataset exceeds 200 TB, so the release includes a 128-hour subset; nuPlan also provides rasterized and vectorized human-annotated semantic maps.Vectorized maps provide more precise geometric information and metadata, while rasterized maps support simplicity and efficient lookup.Source: A. Data collection · p. 3

B. Auto-labeling

nuPlan’s auto-labeling system generates object tracks, infers traffic-light statuses from actor motion, and supports mining a taxonomy of driving scenarios.

B. Auto-labeling

  • Offline perception labels objects without online latency and causality constraints, supporting planning evaluation with very limited perception noise.Its three stages are 3D detection, offline tracking using past and future detections, and global refinement of vehicle-box attributes.Source: B. Auto-labeling · p. 3
  • Traffic-light statuses are inferred from agent motion: movement through an intersection indicates green, while slowing or stationary agents approaching it indicate red.The system covers lanes with observed agents and uses tracks and map information.Source: B. Auto-labeling · pp. 3–4
  • A taxonomy and mining algorithms use low-level attributes, including vehicle speed and state transitions, to identify 73 scenario types beyond monotonous lane-following cases.The attributes come from offline perception tracks and traffic-light statuses.Source: B. Auto-labeling · p. 4

IV. SIMULATION

nuPlan simulates planning from real-world observations through agent prediction, ego planning, and control, in either playback or closed-loop settings. Its agent, controller, and evaluation components support testing planner behavior under different conditions.

IV. SIMULATION

  • The framework initializes from real-world observations, predicts agent trajectories, plans an ego route, and uses a controller to produce a feasible trajectory in open-loop or closed-loop simulation.Open-loop replays recorded actions; closed-loop incorporates the ego vehicle’s actions and can deviate from the recording.Source: §IV · p. 4

A. Agents

  • Reactive agents interact with the ego vehicle and one another, but only vehicles use the reactive IDM policy; vulnerable road users are replayed from logs.This choice reflects the stated difficulty of modeling VRU behavior, which is often uncooperative and hard to model.Source: A. Agents · p. 4

B. Controller

  • Planners provide timed SE(2) pose sequences without a kinematic-feasibility requirement, so a controller tracks them and propagates the state with a motion model.The two-stage implementation uses an LQR tracker and a forward-integrated kinematic bicycle model.Source: B. Controller · p. 4

C. Evaluation

  • Open-loop uses trajectory-error metrics, while closed-loop scores lawfulness, progress, and comfort; normalized metric scores are combined per scenario and averaged across scenarios.The metrics are normalized to [0−1], with higher scores indicating better performance.Source: C. Evaluation · pp. 4–5

scenario score =

The experiments compare planning baselines across challenge setups and examine perception quality and city generalization. Learned planners excel in open-loop, while rule-based planners retain the closed-loop advantage, subject to a stated evaluation caveat.

scenario score =

  • Each challenge score is the average scenario score across all scenarios for that challenge.Source: scenario score = · p. 5

V. EXPERIMENTS

  • The experiments analyze the effects of lower-quality perception inputs and city generalization, and discuss state-of-the-art submissions to the first nuPlan challenge.Source: §V · p. 5

A. Planning baselines

  • Baselines range from constant-speed Simple and map-search/IDM control to raster and vector-attention machine-learning planners trained by imitation.UrbanDriver training also uses agent and trajectory augmentation to mitigate distribution drift in closed-loop simulation.Source: A. Planning baselines · p. 5

B. Main results

  • Supervised-learning planners excel open-loop, but sufficiently sophisticated rule-based planners still outperform purely learned planners closed-loop.IDM achieved the highest closed-loop scores, though its similarity to reactive-agent behavior may give it an unfair advantage.Source: Table III · p. 5, B. Main results · p. 5

C. Perturbation

The experiments examine planner robustness to detection noise and geographic shift, while the nuPlan challenge compares learned, rule-based, and hybrid approaches. The challenge found rule-based planners outperformed purely ML-based planners, with hybrid planners showing the most promise.

C. Perturbation

  • Noise-injected training data reduced UrbanDriver's performance on both nominal and injected closed-loop reactive simulations.The authors interpret this deterioration as indicating the value of high-quality offline annotations in the dataset and learning pipeline.Source: C. Perturbation · pp. 5–6, Table IV · p. 6

D. Generalization

  • Training UrbanDriver only on Las Vegas led to performance drops of 53.8% open-loop, 35.1% closed-loop non-reactive, and 41.5% closed-loop reactive across other locations.Singapore performed worst; the authors attribute this to left-hand traffic versus the right-hand traffic in training, and report a weak correlation between open- and closed-loop performance.Source: D. Generalization · p. 6, Table V · p. 6
  • The authors identify the open-loop-to-closed-loop domain gap as a major challenge to address before broader generalization.Across Singapore, Boston, and Pittsburgh, the open/closed-loop score difference was 16.3%, compared with more than double that, 37.3%, for Las Vegas.Source: D. Generalization · p. 6

E. nuPlan challenge

  • Challenge entrants used supervised learned methods relatively well in open-loop, but closed-loop planners combined learned and handcrafted components.Commonly, a learned model proposed the ego trajectory before refinement for kinematic feasibility and collision avoidance.Source: E. nuPlan challenge · p. 6, Table VI · p. 6

VI. CONCLUSION

  • The first public nuPlan challenge found rule-based planners outperform purely ML-based ones, while hybrid planners show the most promise for difficult scenarios.The highest-scoring closed-loop planner was mostly rule-based and used an ML-generated trajectory only to refine the long-term horizon.Source: §VI · p. 6, E. nuPlan challenge · p. 6

A. Data collection

The data collection combines synchronized, surrounding-view sensors with geographically overlapping but temporally separated dataset splits and planning-oriented maps. The paper also describes object-track statistics and a staged offline autolabeling system that refines detections across tracks.

A. Data collection

  • The fleet's 31 vehicles use five lidars, eight cameras, GNSS, and IMU to cover the full 360-degree environment and minimize blind spots.Lidar and camera timing is aligned; lidar sweeps are merged and transformed to a common frame, with ego-motion compensation.Source: A. Data collection · p. 9, Figure 5 · p. 9, Table VII · p. 9
  • Train, validation, and test splits overlap geographically but exclude data from the same city and day across splits, and are stratified by city, day, and scenario.The semantic maps support a lane-and-connector graph and annotate road features such as stop lines and crosswalks that affect planning.Source: A. Data collection · p. 9, Table VIII · p. 10
  • The rarest of six object classes has more than 105 tracks, although the most common class has about two orders of magnitude more tracks.The authors describe the class distribution as less long-tailed than datasets with more fine-grained classes; box sizes are stable except for irregularly shaped construction vehicles, most objects move slowly, and most vehicle orientations are multiples of ±90°.Source: A. Data collection · p. 9, Figure 6 · p. 10, Figure 7 · p. 11

B. Autolabeling

  • The detector uses seven ego-motion-compensated past lidar frames, points from four side lidars, a cylindrical MVF++ view, a RegNet backbone, and a CenterPoint head.Past and future detections are majority-voted to adjust flipped box headings; adding future input frames did not improve detection.Source: B. Autolabeling · p. 10
  • Offline tracking uses a Kalman-filter multi-object tracker that backdates confirmed tracks and retains them longer before coasting, reducing false negatives and ID switches.These choices use past and future frames and trade slower processing for fewer ID switches.Source: B. Autolabeling · p. 10
  • A single refinement network uses tracks and aggregated point clouds to refine box position, heading, size, and velocity, and is applied to vehicles.Unlike approaches using separate networks for static versus dynamic objects or different box properties, this design reduces deployment overhead; deployment uses median track size and updates position and velocity from network outputs.Source: B. Autolabeling · pp. 10–11

C. Evaluation of the autolabeling system

The autolabeling evaluation compares the offline detection and tracking stages with modified onboard-oriented baselines, then assesses global track refinement. The reported results favor the proposed offline components, including fewer tracking ID switches and improved vehicle-box F1 scores.

Evaluation setup

  • The evaluation uses 3,098 human-labeled scenes from four cities (2,958 training, 49 validation, 91 test); detection and global track refinement use maximum F1, while offline tracking uses AMOTA, ID switches, and recall.Source: C. Evaluation of the autolabeling system · p. 11

3D object detection

Offline tracker

  • The offline tracker reduced ID switches by 78% compared with modified AB3DMOT, with significant improvement on all reported tracking metrics.The authors attribute the ID-switch reduction to the enlarged Agemax and say fewer switches indicate less fragmented tracks that better reflect real-world agent movement.Source: C. Evaluation of the autolabeling system · p. 12, Table X · p. 12

Global track refinement

D. Traffic lights

Traffic-light labels are inferred from agent motion relative to mapped lane connectors and then post-processed to address shared directions and transition lag. Against YOLOv3, the system recovered more statuses and had slightly higher classification accuracy among visible lights, though transitions remained difficult.

Traffic-light status inference

  • The system infers green from agents moving along the mapped lane connector and red from agents stopped or decelerating near the intersection; unobserved lanes are labeled unknown.Green and amber are treated alike because vehicles may proceed under either status; inference uses past and future trajectories.Source: D. Traffic lights · p. 13

Post-processing

  • Post-processing shares statuses across parallel lane connectors and back-fills recent statuses to account for lag between light changes and drivers' movements.The grouping step is intended to reduce false negatives where some parallel lanes lack observed agents.Source: D. Traffic lights · p. 13

Evaluation

  • The motion-inference system recovered 5.2x more traffic-light statuses than YOLOv3 and achieved 68.7% accuracy among visible lights, versus YOLOv3's 66.4%.The comparison manually labeled about 1,000 frames; transitions were difficult for motion inference because drivers react with a lag after signal changes.Source: D. Traffic lights · p. 13

E. Scenario mining

Scenario mining produces high-precision labels across 73 scenario types, while the accompanying agent and evaluation methods address multi-agent interactions and compare planners across open- and closed-loop settings.

E. Scenario mining

  • Scenario mining combines atomic primitives into SQL queries, adds 10 seconds before and after each match, and manually checks 100 examples per scenario.The process yields 73 scenario types; it prioritizes precision over recall, and Fig. 11 shows their distributions by city.Source: E. Scenario mining · p. 13, Figure 11 · p. 14

A. Agents

  • The IDM lead-object search filters for objects that intersect the agent’s planned path, avoiding nearby objects in adjacent lanes.The baseline is expanded into a polygon along the path and as wide as the agent.Source: A. Agents · p. 14, Figure 12 · p. 14
  • Extending object footprints in proportion to speed can help IDM handle merges by identifying an oncoming agent earlier and prompting the agent to slow down.IDM does not inherently account for multi-lane interactions.Source: A. Agents · pp. 14–15
  • LaneGCN’s independent agent predictions can produce colliding trajectories, while closed-loop distribution shift can induce unrealistic scenarios.The simulated agent count also had to be limited to maintain acceptable runtime.Source: A. Agents · p. 15

B. Evaluation

  • Open-loop errors and miss rate are averaged across 3, 5, and 8-second horizons sampled at 1 Hz, supporting comparison across different planning horizons or sampling rates.Within-bound scores are zero when average error exceeds thresholds of 8 m for distance, 0.8 rad for heading, or 0.3 for miss rate.Source: B. Evaluation · p. 15
  • Closed-loop scoring assigns zero for specified safety or progress violations, while one object collision or more than 2 m of oncoming travel can halve the weighted score.The scoring heuristic is an initial proposal hand-tuned for the dataset and simulation framework.Source: B. Evaluation · p. 16

A. Planning experiments

Submitted planners commonly scored higher in the open-loop challenge than in closed-loop challenges. Scenario-stratified results also indicate that the metrics can favor conservative, rules-abiding driving over human-like behavior.

A. Planning experiments

  • Scenario-stratified metrics show that planners avoiding collisions and staying within the drivable area may outscore planners attempting to mimic human drivers.The authors conclude that finer-grained, scenario-based metrics are needed to distinguish rules-abiding driving from desirable human-like behavior.Source: A. Planning experiments · p. 16, Table XV · p. 17

B. nuPlan Challenge

Loading 2403.04133v1…