Source-linked AI summary
ExceptionDrive: A Planning-Oriented Counterfactual Corner-Case Benchmark for Autonomous Driving
Ziyi Luo, Zhe Sun, Yehao Lu, Lei Zhou, Lisheng Wu, Xuewei Li, Zequn Qin, Xi Li
TL;DR
Routine-driving benchmarks do not establish reliability under rare, safety-critical hazards, and inserting hazards can invalidate recorded human trajectories. ExceptionDrive inserts localized hazards into real nuScenes scenes and evaluates edited predictions without expert references, while a Reminder Agent converts visual evidence into structured high-level guidance. Representative planners frequently intrude into hazards or provide insufficient clearance, and reminders improve zero-shot strategy accuracy while reducing under-warning.
Problem
Routine-heavy benchmarks and limited controllable real-world hazards leave planner reliability under rare, safety-critical scenarios insufficiently evaluated.
Source: §1 · p. 2Method
ExceptionDrive inserts localized hazards into real nuScenes scenes and evaluates edited trajectories with task-specific constraints and reference-free safety metrics.
Source: §1 · pp. 2–3Results
Representative planners frequently intrude into hazards or provide insufficient clearance, while zero-shot reminders improve strategy accuracy and reduce under-warning.
Source: §1 · p. 3Takeaways & Limitations
The benchmark tests local hazard compliance without relying on expert trajectories that may be invalid after hazard insertion.
Source: §1 · p. 3CTS measures sensitivity to the counterfactual hazard but does not determine whether the resulting response is safe or directionally appropriate.
Source: §3.3 · p. 7
Abstract
from arXiv · showhide
Average performance on routine driving benchmarks does not establish planner reliability under rare, safety-critical hazards. We proposed ExceptionDrive, a counterfactual planning benchmark that uses VLM-assisted screening, localized multi-view editing, and quality auditing to insert hazards into real nuScenes scenes while preserving their context. Its 21 tasks span six safety families and define hazard or conflict regions, local safety constraints, and acceptable responses. Because hazard insertion can invalidate the recorded human trajectory, our reference-free protocol evaluates edited predictions using Unsafe Rate (UR), Hazard Clearance Compliance (HCC), Hazard Proximity Response (HPR), and Counterfactual Trajectory Shift (CTS), which measure core-region intrusion, clearance compliance, clearance relative to a prescribed margin, and counterfactual trajectory change. Seven representative planners frequently intrude into hazard regions or provide insufficient clearance. We also develop a Reminder Agent that, without sample-specific task labels, converts visual evidence and the shared taxonomy into structured records of hazard presence, type, and a recommended high-level strategy. The agent neither predicts trajectories nor controls the vehicle; its records guide a VLM-based decision agent. In zero-shot experiments, the reminders improve strategy accuracy and reduce under-warning.
1 Introduction
Routine-driving benchmarks leave planner reliability under rare, safety-critical hazards unestablished, while existing resources offer limited controlled evaluation and hazard insertion can invalidate human-trajectory references. ExceptionDrive addresses this gap with localized edits to real scenes, task-specific safety constraints and acceptable responses, reference-free evaluation, and an auxiliary Reminder Agent.
1 Introduction
- The benchmark combines VLM-assisted scene screening, localized multi-view editing, quality review, and BEV hazard localization to create controlled hazards in real scenes.The paired source and edited observations use identical model settings and non-visual inputs, isolating the effect of the localized edit.Source: §1 · p. 2
- ExceptionDrive evaluates 21 localized-hazard tasks across six safety families, with local constraints and acceptable responses instead of a single prescribed trajectory.It inserts hazards into real nuScenes observations while preserving surrounding context as much as possible.Source: §1 · pp. 2–3
- Its reference-free protocol measures core-region intrusion, clearance compliance, hazard-proximity response, and counterfactual trajectory change.Hazard insertion can make the recorded human trajectory invalid, and different responses—including slowing, stopping, yielding, or bypassing—may be acceptable under different constraints.Source: §1 · pp. 2–3
- Representative planners frequently intrude into hazards or provide insufficient clearance, while structured Reminder Agent records improve zero-shot strategy accuracy and reduce under-warning.The agent maps visual evidence and the shared taxonomy to a high-level strategy recommendation without predicting trajectories or controlling the vehicle.Source: §1 · p. 3
2 Related Work
Related work covers VLM-assisted generation and assessment, corner-case datasets, and simulation benchmarks. ExceptionDrive uses VLMs to select and edit real scenes, then complements existing settings with paired-scene planner testing.
2.1 VLM-Assisted Data Generation for Autonomous Driving
- ExceptionDrive uses VLMs to select task-compatible real scenes, generate location-aware editing instructions, and review edits rather than synthesize entire scenes.Keeping each source scene as a within-scene control is intended to isolate the inserted hazard.Source: §2.1 · p. 3
2.2 Corner-Case Taxonomies and Dataset Construction
- Existing corner-case resources target perception, event or semantic reasoning, and interactive simulation, while ExceptionDrive tests planner behavior on paired real scenes.Real-world data offer visual realism but limited control over rare hazards; simulation offers control but has a domain gap.Source: §2.2 · pp. 3–4
3 Method
ExceptionDrive specifies localized hazards, safety constraints, and acceptable responses, then constructs paired edited scenes and evaluates planner predictions against task-defined hazard regions. A task-grounded Reminder Agent turns visual risk evidence into structured records to support high-level strategy selection.
3.1 Task Taxonomy
- The taxonomy groups 21 tasks into six planning-safety families, specifying scene suitability, hazard regions, local constraints, and multiple acceptable risk-mitigation responses.Options include avoiding a risk region, stopping before a boundary, yielding, or bypassing when sufficient space is available.Source: §3.1 · pp. 4–5
3.2 Dataset Construction
- Task specifications guide scene screening, localized multi-view hazard editing, quality review, and BEV hazard-region localization while preserving the surrounding scene.Review checks task validity and visual quality; each retained sample receives a hazard or conflict region for evaluation.Source: Figure 2 · p. 4, §3.2 · p. 5
3.3 Paired Counterfactual Planner Evaluation
- The evaluation uses task-defined local safety constraints and acceptable responses rather than a unique expert trajectory, because hazard insertion can invalidate the recorded human path.UR measures core-region intrusion, HCC clearance compliance, HPR clearance relative to the prescribed margin, and CTS counterfactual trajectory change.Source: §3.3 · pp. 5, 7
- HCC and UR are the primary safety metrics, while HPR and CTS provide complementary diagnostics rather than standalone monotonic safety scores.HPR diagnoses whether safe avoidance is near the minimum sufficient clearance; CTS measures counterfactual sensitivity but does not determine whether a response is safe or directionally appropriate.Source: §3.3 · p. 7
3.4 Task-Grounded Risk Reminder
- The Reminder Agent converts task-relevant visual evidence into a structured risk record and supports a VLM-based agent’s high-level driving strategy selection without predicting trajectories or controlling the vehicle.It has access to taxonomy risk definitions but not the ground-truth task identity or target strategy for the sample.Source: §3.4 · p. 8
4 Experiments
Experiments assess ExceptionDrive’s benchmark quality, planner safety behavior across corner cases, and task-grounded reminders for zero-shot VLM decision making. Planners often fail local safety checks, while reminders improve strategy prediction.
4.1 Dataset Statistics and Quality Analysis
- Representative examples pair original and edited scenes, with localized hazards inserted while surrounding scene content is largely preserved.The samples span six safety families and retain viewpoint, road layout, illumination, weather, and surrounding traffic context.Source: Figure 4 · p. 8, §4.1 · p. 9
- The final benchmark retains 3,181 edits at a 92.31% acceptance rate; manual inspection confirms 163 of 168 audited VLM-approved samples.The collection also includes 9,259 auxiliary pairs, which are not used for reported benchmark evaluation.Source: §4.1 · pp. 8–9
4.2 Planner Evaluation on Corner Cases
- 21.03% is the highest HCC across planners, and every planner’s UR exceeds 78%; ST-P3 leads on both safety metrics, while DrivoR performs worst on both.Impromptu VLA has the largest CTS (2.20) without the best safety performance, so trajectory shift alone does not indicate safe avoidance.Source: Table 1 · p. 9, §4.2 · p. 9
- ST-P3 has the highest HCC in most safety families, while DrivoR reaches 2.61% on Bypass and Gap Conflict.For Signal and Rule Conflict and Intersection Conflict, HCC measures compliance with task-defined regions rather than complete traffic-rule understanding.Source: Table 2 · p. 9, §4.2 · pp. 9–10
4.1 Dataset Statistics and Quality Analysis
- Signal and Rule Conflict is the least frequent family at 3.60%, reflecting its specific intersection layouts, traffic-light states, and right-of-way contexts.The six families account for 25.56%, 19.75%, 24.96%, 3.60%, 9.10%, and 17.03%, respectively, in the order reported.Source: §4.1 · p. 9
4.3 Qualitative Analysis
- Qualitative cases show planners may enter the risk region, shift in an unsafe direction or insufficient amount, or avoid intrusion while remaining too close to the boundary.Intrusion, insufficient clearance, and trajectory change are related but non-equivalent phenomena.Source: Figure 5 · p. 10, §4.3 · p. 10
4.4 Reminder Agent Evaluation
- The Reminder Agent raises zero-shot strategy accuracy from 51.86% to 70.71% and reduces miss rate from 45.53% to 29.18%.The evaluation compares direct scene-based decisions with decisions receiving a structured reminder, while keeping visual inputs and candidate strategies identical.Source: §4.4 · pp. 10–11, Table 3 · p. 11
5 Conclusion
ExceptionDrive evaluates planners under localized hazards in real driving observations, using a hazard-centric protocol where hazard insertion invalidates conventional trajectory references. Experiments show frequent clearance failures by planners and improved zero-shot strategy prediction with the Reminder Agent.
5 Conclusion
- ExceptionDrive provides 21 tasks across six planning-safety families by inserting localized hazards into compatible real-world observations while preserving scene context.The benchmark targets rare, safety-critical local hazards.Source: §5 · p. 11
- Its reference-free protocol assesses intrusion, clearance compliance, avoidance response, and counterfactual behavioral change after hazard insertion.The protocol addresses the absence of valid expert trajectories in edited hazardous scenes.Source: §5 · p. 11
- Representative planners frequently fail to maintain sufficient clearance, while the Reminder Agent improves zero-shot strategy prediction and reduces under-warning.The paper identifies systematic analysis of planner behavior under localized visual interventions as a value of ExceptionDrive.Source: §5 · p. 11