PaperlayerView exact revision on arXiv

Source-linked AI summary

NavSafe-$\infty$: Benchmarking Closed-Loop Driving Safety in Photorealistic Environments

Yuxin Bao, Hongwei Ruan, Luobin Wang, Seth Z. Zhao, Ziyang Leng, Zihan Zhang, Yu Zeng, Rowan McAllister, Henrik Christensen, Bolei Zhou

arXiv:2609.26618v1cs.RO

TL;DR

Open-loop driving benchmarks score static trajectories, leaving compounding errors and reactive interactions untested. NAVSAFE-∞ evaluates event-specific safety in photorealistic closed loop; its results show that open-loop strength does not reliably transfer to closed-loop safety.

  • Problem

    Static open-loop evaluation cannot capture sequential policy–environment interactions, compounding errors, or reactive traffic, limiting evidence about policy reliability in safety-critical situations.

    Source: §1 · p. 2

  • Method

    NAVSAFE-∞ evaluates policies in photorealistic closed-loop scenarios with event-specific objectives and success and failure criteria across a structured traffic-safety taxonomy.

    Source: §1 · p. 2, §3 · p. 4, §3.1 · p. 4

  • Results

    Strong open-loop performance does not reliably translate into closed-loop safety, and common post-imitation-learning remedies do not reliably resolve the observed failures.

    Source: §1 · p. 2, §7 · p. 10

  • Takeaways & Limitations

    Event-level closed-loop evaluation reveals specific behavioral failures that non-reactive evaluations can obscure.

    Source: §7 · p. 10

    The benchmark primarily covers dense urban driving, with highway, rural, adverse-weather, and nighttime conditions comparatively underrepresented.

    Source: §6 · p. 9

Abstract

from arXiv · show

End-to-end (E2E) driving policies have progressed rapidly on open-loop (OL) benchmarks, yet OL evaluation cannot reveal whether a policy withstands compounding errors, recovers from failures, or interacts safely with surrounding actors. We introduce NavSafe-$\infty$, a photorealistic closed-loop (CL) benchmark of 280 scenarios spanning 28 event types, each with success and failure criteria defined within a structured traffic-safety taxonomy, which yields category-level capability scores for Traffic Crashes, Vulnerable Road User Crashes, Traffic Violations, and Traffic Incidents. Evaluating 20 E2E policies, we find that OL gains do not reliably transfer to CL safety. Analyzing two common remedies further shows that passive demonstration perturbation helps mainly when CL rollouts stay near its perturbed training states, and that OL reinforcement-learning fine-tuning exhibits reward hacking by trading safety margin for ego progress, which CL feedback amplifies into compounding safety-critical errors. Together, these results demonstrate the blind spot of OL benchmarks indicating CL safety success. The benchmark and an extensible toolbox for customizable event curation and policy diagnosis will be open-sourced and maintained to facilitate future research.

1 INTRODUCTION

Static open-loop evaluation does not show whether driving policies remain safe through sequential interactions, compounding errors, or recovery from failures. NAVSAFE-∞ introduces event-level closed-loop evaluation and uses it to examine transfer from open-loop performance and the effects of common mitigation methods.

Introduction

  • Open-loop evaluation scores predicted trajectories without sequential policy–environment feedback, leaving compounding errors and reactive interactions untested.Policies with similar open-loop scores can reach different closed-loop safety outcomes in the same scenario.Source: §1 · p. 2
  • NAVSAFE-∞ evaluates policies in photorealistic closed-loop scenarios with event-specific success and failure conditions derived from real-world traffic-safety events.Its event-level evaluation identifies specific safety failures and assesses whether open-loop gains translate to closed-loop safety.Source: §1 · p. 2
  • Strong open-loop performance does not reliably transfer to closed-loop safety, and high-ranking policies remain vulnerable across multiple safety-critical event types.Passive perturbation helps mainly when rollouts stay near perturbed training states; RL fine-tuning can reward-hack the open-loop score, with closed-loop feedback amplifying errors.Source: §1 · p. 2
  • The paper contributes an extensible toolbox for customizable scenario curation and policy diagnosis under controlled changes to ego and environment states.The toolbox is intended to support adding safety-critical events and diagnostic protocols.Source: §1 · p. 2

2 RELATED WORK

Prior driving benchmarks either score static logged states or execute policies in closed loop without a shared, fine-grained safety taxonomy. Related policy work spans offline imitation learning, offline reinforcement learning, passive demonstration perturbation, and world-model-based approaches.

2.1 END-TO-END DRIVING BENCHMARKS

  • Open-loop benchmarks score trajectories or planning behavior on logged observations, but cannot expose compounding error, recovery, or reactive traffic, and their rankings track closed-loop performance only loosely.Examples include nuScenes, NAVSIM, WOMD-Reasoning, and nuReasoning.Source: §2.1 · p. 2
  • Existing closed-loop benchmarks provide sequential feedback, but use different scenarios and aggregate scores rather than a fixed traffic-safety taxonomy and ability-level reporting.Some use rendered imagery with a domain gap for real-sensor policies, while newer systems add photorealistic scenes or reconstructed renderers.Source: §2.1 · p. 3

2.2 END-TO-END DRIVING POLICIES

  • Most imitation-learning and supervised-fine-tuning policies learn from fixed demonstrations and inherit covariate shift because training omits policy-induced test states.Offline reinforcement learning on logged observations likewise leaves reward-exploiting policies uncorrected by environmental feedback.Source: §2.2 · p. 3
  • Passive perturbation methods synthesize states absent from logs, while world-model-based policies combine future prediction with planning.SimScale perturbs ego trajectories in reconstructed reactive scenes; BeyondDrive mines expert-proximate unsafe trajectories, and world-model examples include DriveLaW, DriveVLA–W0, and SimWAM.Source: §2.2 · p. 3

3 NAVSAFE-∞BENCHMARK

NAVSAFE-∞ is a photorealistic, event-based closed-loop benchmark designed to assess specific safety capabilities while retaining policy–environment feedback. It defines event-specific tasks and metrics, and reports improved rendering quality over existing generative renderers on three metrics.

3 NAVSAFE-∞BENCHMARK

  • The benchmark comprises 280 event-based scenarios across 28 event types adapted from RoadSafe365, organized into four traffic-safety categories.The categories are Traffic Crashes, Vulnerable Road User Crashes, Traffic Violations, and Traffic Incidents.Source: §3 · p. 4

3.1 EVENT-BASED BENCHMARK DESIGN

  • Bounded event-centered scenarios isolate specific safety capabilities while preserving closed-loop feedback throughout each rollout.An event specifies a bounded traffic interaction or rule-governed situation, with an initial scene, relevant road users or controls, task requirements, and termination conditions.Source: §3.1 · p. 4, Figure 3 · p. 4

3.2 EVALUATION PROTOCOL AND METRICS

  • The protocol reports Driving Score, Success Rate, Driving Efficiency, Comfort, and category-level Ability Scores computed as unweighted means of event-type success rates.Driving Score combines event progress with penalties for recorded infractions; Success Rate requires meeting event-specific criteria within the time budget without a disallowed infraction.Source: §3.2 · p. 5

3.3 RENDERING FIDELITY AND POLICY-RELEVANT VISUAL ALIGNMENT

  • Rendering fidelity is assessed against real camera frames using FID, FVD, and a policy-oriented Fréchet-distance metric.The policy-oriented FDk π averages scene-context feature distances across k E2E policies and normalizes by the trace of real-feature covariance.Source: §3.3 · p. 5
  • NAVSAFE-∞ achieves better rendering quality than existing generative renderers on all three reported metrics.The comparison uses rendered observations against corresponding recorded clips; the long-horizon illustration shows temporal consistency for NAVSAFE-∞, unlike temporal inconsistency or cumulative drift in other renderers.Source: §3.3 · p. 5, Table 1 · p. 5, Figure 4 · p. 5

4 EXPERIMENTS

The experiments investigate open-loop to closed-loop transfer, unsafe-behavior mechanisms, and inconsistent gains from post-imitation-learning methods. The leaderboard is organized around global closed-loop metrics and category-level ability scores.

4 EXPERIMENTS

  • The experiments ask whether NAVSIM open-loop performance is associated with closed-loop success, what mechanisms underlie unsafe outcomes, and why perturbation and RL fine-tuning yield inconsistent gains.Source: §4 · p. 6

4.1 NAVSAFE-∞LEADERBOARD

  • Policy evaluation uses four global closed-loop metrics and four category-specific ability scores.The complete leaderboard is deferred to the appendix.Source: §4.1 · p. 6
  • The leaderboard reports policy venue, closed-loop metrics, and closed-loop ability scores, with colors marking the top three learned-policy point estimates in each column.Its ± values are sample standard deviations across three per-seed estimates.Source: Table 2 · p. 6, §4.1 · p. 6

DS SR DE Comf. TC VRUC TV TI

The section diagnoses distinct closed-loop safety failures and shows why passive perturbation and open-loop reinforcement-learning fine-tuning do not consistently improve closed-loop outcomes.

4.2 FAILURE INVESTIGATIONS

  • LTF detects crossing traffic but continues nominal car-following behavior, delaying braking until collision is unavoidable.The failure is consistent with causal confusion: routine driving correlations govern planning despite the detected hazard.Source: §4.2 · p. 7, Figure 5 · p. 7
  • DrivoR has a collision-free stopping candidate, but its scorer ranks the collision-bound proposal higher.This localizes the observed failure to trajectory ranking rather than candidate generation.Source: §4.2 · p. 7, Figure 5 · p. 7
  • SimWAM omits an oncoming vehicle from its future prediction, removing a safety-critical constraint and leading the plan through its occupied space.The failure concerns action-relevant actor persistence in the predicted world state.Source: §4.2 · p. 7, Figure 6 · p. 7
  • AutoVLA calls for stopping in two policy-induced states, yet only one resulting trajectory stops, exposing unstable reasoning–action alignment.The matched comparison attributes the divergence to reasoning–action grounding across states.Source: §4.2 · pp. 7–8, Figure 6 · p. 7

4.3 UNDERSTANDING THE LIMITS OF PASSIVE DEMONSTRATION PERTURBATION AND RLFT IN CLOSED-LOOP

  • Perturbation helps when rollouts stay near its training-state support, while open-loop reward hacking can carry into closed-loop execution and compound safety errors.The perturbation outcome depends on rollout coverage; reward exploitation trades safety margin for progress.Source: §4.3 · p. 8, Figure 7 · p. 8
  • Open-loop RLFT can trade safety margin for progress, and closed-loop feedback amplifies resulting errors into degraded safety performance.Figure 8 describes early rollout termination in safety-critical scenarios as a source of amplified degradation.Source: §4.3 · p. 8, Figure 8 · p. 9

5 NAVSAFE-∞TOOLBOX

The toolbox supports photorealistic closed-loop environment construction, customizable safety scenarios, and diagnosis through controlled replay of policy-induced states.

5 NAVSAFE-∞TOOLBOX

  • The environment tool converts real-world driving logs into interactive simulation environments with reconstructed 3D Gaussian Splatting scenes.It integrates sensor streams, ego poses, actor tracks, traffic-light states, and map context.Source: §5 · p. 9
  • The event-design tool lets researchers add assets and specify their size, pose, and behavior to create new safety events or controlled variants.Behaviors can be static, scripted, or reactive.Source: §5 · p. 9
  • The diagnosis tool archives policy-induced states for counterfactual replays that separately intervene on ego motion or surrounding-actor behavior.These replays test whether improvements persist as rollouts move beyond logged or perturbed demonstration states.Source: §5 · p. 9

6 LIMITATIONS

The authors bound their conclusions to the evaluated simulation and note that the benchmark underrepresents several driving conditions.

6 LIMITATIONS

  • Photorealistic rendering narrows the visual-domain gap but does not capture all sensor noise, reconstruction artifacts, contact dynamics, or human behavior.Results are controlled evidence in the evaluated simulation, not direct estimates of real-world crash risk.Source: §6 · p. 9
  • The benchmark focuses mainly on dense urban driving, with highway, rural, adverse-weather, and nighttime conditions comparatively underrepresented.The authors identify broader environmental coverage as a direction for future evaluation.Source: §6 · p. 9

7 CONCLUSION

The conclusion argues that driving safety should be evaluated through policy-induced interactions and summarizes the benchmark’s diagnosed failure modes and remedy limits.

7 CONCLUSION

  • The conclusion calls for assessing E2E driving safety through policy-induced interactions, using photorealistic event-level closed-loop evaluation.It says these evaluations reveal behavioral failures that non-reactive evaluations can obscure.Source: §7 · p. 10
  • The diagnosed failures include causal confusion, scorer bottlenecks, latent dynamic discrepancy, and reasoning–action misalignment.The analysis concerns failures under closed-loop execution.Source: §7 · p. 10
  • The conclusion says perturbation helps mainly near its support, while reward hacking with RLFT carries into closed loop and compounds through feedback.The authors state that common post-imitation-learning remedies do not reliably resolve the failures.Source: §7 · p. 10
Loading 2609.26618v1…