Source-linked AI summary
NAVSIM: Data-Driven Non-Reactive Autonomous Vehicle Simulation and Benchmarking
Daniel Dauner, Marcel Hallgarten, Tianyu Li, Xinshuo Weng, Zhiyu Huang, Zetong Yang, Hongyang Li, Igor Gilitschenski, Boris Ivanovic, Marco Pavone, Andreas Geiger, Kashyap Chitta
TL;DR
Autonomous-driving benchmarks struggle to reflect closed-loop performance while remaining scalable and close to real-world data. NAVSIM combines large real-world datasets with a short-horizon non-reactive simulator and simulation-based metrics, and finds that TransFuser can match much larger end-to-end architectures such as UniAD on challenging scenarios.
Problem
Existing open-loop metrics can misrepresent safe trajectories, while closed-loop simulation is difficult to scale and can exhibit a domain gap to real-world driving.
Method
NAVSIM combines curated challenging real-world scenarios with non-reactive bird’s-eye-view simulation and configurable metrics for trajectory evaluation.
Results
On NAVSIM’s challenging scenarios, TransFuser and PARA-Drive achieve a PDMS of 84.0, while UniAD achieves 83.4.
Takeaways & Limitations
NAVSIM provides an accessible toolkit intended to bridge simulated and real-world driving evaluation through standardized scenarios and metrics.
Takeaways & Limitations
NAVSIM does not model reactiveness or compounding closed-loop errors, so a high PDMS does not always imply a high CLS.
Abstract
from arXiv · showhide
Benchmarking vision-based driving policies is challenging. On one hand, open-loop evaluation with real data is easy, but these results do not reflect closed-loop performance. On the other, closed-loop evaluation is possible in simulation, but is hard to scale due to its significant computational demands. Further, the simulators available today exhibit a large domain gap to real data. This has resulted in an inability to draw clear conclusions from the rapidly growing body of research on end-to-end autonomous driving. In this paper, we present NAVSIM, a middle ground between these evaluation paradigms, where we use large datasets in combination with a non-reactive simulator to enable large-scale real-world benchmarking. Specifically, we gather simulation-based metrics, such as progress and time to collision, by unrolling bird's eye view abstractions of the test scenes for a short simulation horizon. Our simulation is non-reactive, i.e., the evaluated policy and environment do not influence each other. As we demonstrate empirically, this decoupling allows open-loop metric computation while being better aligned with closed-loop evaluations than traditional displacement errors. NAVSIM enabled a new competition held at CVPR 2024, where 143 teams submitted 463 entries, resulting in several new insights. On a large set of challenging scenarios, we observe that simple methods with moderate compute requirements such as TransFuser can match recent large-scale end-to-end driving architectures such as UniAD. Our modular framework can potentially be extended with new datasets, data curation strategies, and metrics, and will be continually maintained to host future challenges. Our code is available at https://github.com/autonomousvision/navsim.
1 Introduction
NAVSIM addresses shortcomings in autonomous-driving evaluation by combining challenging real-world scenarios, simulation-based metrics, and non-reactive simulation. It aims to support more principled and standardized comparisons of driving policies.
- Evaluation gaps: Existing benchmarks often contain trivial scenes, allowing policies that extrapolate the vehicle’s past trajectory to achieve strong performance.These datasets prioritize visual diversity and label quality rather than planning relevance.
- Evaluation gaps: Average displacement error can misrepresent trajectory quality because safe trajectories may legitimately diverge from recorded human driving logs.Driving evaluation must balance potentially conflicting objectives such as safety, comfort, and progress.
- Evaluation gaps: Existing simulators support interactive evaluation but face realism and scalability challenges, including domain gaps from synthetic sensor data.The lack of standardized evaluation setups also creates metric inconsistencies and unfair comparisons.
- NAVSIM approach: NAVSIM samples over 100k challenging real-world driving scenarios and uses a non-reactive bird’s-eye-view simulation to avoid inaccurate sensor simulation.The evaluated agent commits to actions for a fixed horizon while other agents’ future behavior is assumed unaffected.
- NAVSIM approach: NAVSIM combines standardized data splits, configurable simulation-based metrics, and reproduced end-to-end baselines for independent benchmarking.Its framework includes data curation tools and an official public evaluation server.
2 Related Work
Prior work divides autonomous-driving evaluation between closed-loop simulation and open-loop displacement-error benchmarks. Each paradigm offers benefits but retains realism, scalability, or comparability limitations.
- Closed-loop benchmarking: Closed-loop simulators collect downstream statistics such as collisions, traffic-rule compliance, and comfort during interactive evaluation.Examples include CARLA, Metadrive, nuPlan, and Waymax.
- Closed-loop benchmarking: Realistic simulation of traffic behavior and sensor data remains challenging, limiting the reliability or scalability of closed-loop evaluation.Established simulators commonly rely on simulated camera or LiDAR inputs.
- Open-loop benchmarking: Open-loop benchmarks commonly compare predicted trajectories with recorded expert trajectories using displacement errors.This approach is easy to apply to recorded driving data but does not directly evaluate downstream driving behavior.
- Open-loop benchmarking: The absence of standardized planning metrics has led to independent implementations and inconsistencies in reported or compared results.These issues have been documented particularly for nuScenes-based planning evaluation.
3 NAVSIM: Non-Reactive Autonomous Vehicle Simulation
NAVSIM evaluates trajectory-planning agents with a short-horizon, non-reactive simulation over curated challenging scenes. Its configurable score combines safety-related penalties with progress, collision-risk, and comfort objectives.
- Task description: NAVSIM agents plan future poses over h seconds from sensor streams, ego status, and a one-hot left, straight, or right navigation goal.Ego status includes current speed and acceleration.
- Non-Reactive Simulation: The non-reactive simulator queries each agent only in the initial scene frame and keeps its planned trajectory fixed for the full trajectory duration.This avoids repeatedly simulating high-dimensional sensor streams at closed-loop frequencies.
- PDM Score: NAVSIM computes subscores after simulation and aggregates them into the configurable PDM Score in the range [0, 1].The score can be modified by adding or removing subscores and changing aggregation parameters.
- Penalties: Collisions and drivable-area violations impose hard penalties, setting the corresponding subscores to zero and the scene’s PDM Score to 0.The penalties target traffic-rule compliance and the safety of pedestrians and road users.
- Weighted Average: The weighted average accounts for ego progress, time-to-collision, and comfort alongside the safety penalties.Ego progress is measured relative to an approximated safe upper bound from the PDM-Closed planner.
- Weighted Average: The challenge weighting uses weightEP = 5, weightTTC = 5, and weightC = 2, while the top three challenge ranks remain unchanged under equal weighting.The authors describe this selection as reasonable and robust to changes.
- Dataset and filtering: NAVSIM uses OpenScene and filters out near-trivial or erroneous frames to create challenging splits for evaluation.A constant-velocity baseline reaches 79% PDMS on OpenScene, compared with 91% human-level performance.
4 Experiments
The experiments test whether NAVSIM’s non-reactive open-loop evaluation aligns with closed-loop metrics and what it reveals about end-to-end driving models. PDMS aligns better with closed-loop performance than OLS, while TransFuser matches larger architectures on challenging scenarios.
- Alignment Between Open-Loop and Closed-Loop Evaluation: PDMS consistently correlates better with closed-loop performance than OLS across learned and rule-based planners.PDMS is better correlated with CLS in both rank and linear correlation analyses, and remains better for each planner type.
- Alignment Between Open-Loop and Closed-Loop Evaluation: Reducing CLS duration from d = 15s to d = 4s further raises the correlation of PDMS and OLS with closed-loop evaluation.The shorter simulation horizon more closely matches the open-loop counterparts.
- Alignment Between Open-Loop and Closed-Loop Evaluation: Longer PDMS horizons, ranging from h = 2s to h = 8s, increase correlation with closed-loop metrics.The result indicates value in long horizons despite the difficulty of predicting future motion over 8s in uncertain scenarios.
- Analysis of the State of the Art in End-to-End Autonomous Driving: On navtest, all sensor agents exceed 83 PDMS, while TransFuser and PARA-Drive perform best at 84.0.Camera-only LTF reaches 83.8 and UniAD reaches 83.4, without surpassing TransFuser and LTF despite more demanding training.
- Analysis of the State of the Art in End-to-End Autonomous Driving: TransFuser’s PDMS decreases by 1.5−2.6 without velocity and acceleration, and by 1.0−2.1 without acceleration alone.These ablations show that TransFuser benefits from ego-status inputs while not relying purely on kinematic state.
- NAVSIM Challenge: The CVPR 2024 NAVSIM Challenge received 463 submissions from 143 teams through a private-data leaderboard evaluation.Participants submitted trajectories that were simulated and scored using PDMS.
5 Discussion
NAVSIM improves alignment with closed-loop driving but remains limited by its non-reactive evaluation, simplified metrics, and dataset constraints. The authors recommend pairing it with graphics-based closed-loop simulators and expanding its metrics and dataset support.
- NAVSIM does not model reactiveness or compounding error accumulation, so high PDMS does not always imply high CLS.
- Rear-end collisions into the ego vehicle are not classified as at-fault, reducing the importance assigned to scenes behind the vehicle.
- NAVSIM should be used alongside graphics-based closed-loop simulators such as CARLA when developing planning algorithms.
- PDMS and CLS omit some traffic rules and broader concepts, including stop-sign compliance, traffic-light compliance, transit, and fuel efficiency.
- NAVSIM inherits nuPlan dataset issues, including missing label classes, camera-parameter errors, noisy poses and annotations, and absent road-elevation data.