PaperlayerView exact revision on arXiv

Source-linked AI summary

Human Behavior-Informed Crash Scenario Generation with Real-World Crash Priors for Autonomous Vehicle Safety Evaluation

Mingxing Peng, Xusen Guo, Long Chen, Xintao Yan, Siyu Teng, Jun Ma

arXiv:2610.04366v1cs.ROcs.AI

TL;DR

Rare crashes make reliable AV safety evaluation through real-world driving costly, while existing generators often miss realistic pre-impact behavior and crash-type distributions. CrashSim transfers retrieved crash priors into generative traffic simulation; it more closely reproduces real-world crash characteristics, and nuCrash better differentiates planner safety performance than nuScenes.

  • Problem

    Crashes and severe near-crashes are rare, and existing scenario-generation methods often give limited attention to pre-impact interaction processes and real-world crash-type distributions.

    Source: Introduction · pp. 1, 3

  • Method

    CrashSim retrieves contextually relevant real-world crash priors and adapts their motion patterns and behavior-stage timing to guide diffusion-based multi-agent simulation.

    Source: Introduction · p. 3

  • Results

    CrashSim more closely reproduces real-world pre-impact behavior and collision characteristics, while nuCrash more clearly differentiates five planners’ safety performance than nuScenes.

    Source: Introduction · p. 3, Results · p. 4

  • Takeaways & Limitations

    CrashSim transfers sparse crash priors across naturalistic driving scenes to support realistic crash generation and more informative AV safety evaluation.

    Source: Discussion · p. 11

    The LLM-based evaluation associates performance changes with observed behaviors and conditions, but its attribution of underlying failure mechanisms remains relatively coarse.

    Source: Discussion · p. 11

Abstract

from arXiv · show

Reliable safety evaluation of autonomous vehicles (AVs) is essential to improving road safety, yet it depends critically on realistic simulation of rare crashes. Existing crash scenario generation methods can increase collision occurrence, but often fail to realistically reproduce how crashes evolve before impact or the distribution of crash types observed in the real world. Here, we present CrashSim, a human behavior-informed crash scenario generation framework that uses real-world crash priors to guide generative multi-agent traffic simulation for more reliable AV safety evaluation. These priors capture how real-world crashes evolve before impact and how different crash types are distributed, allowing limited crash data to guide realistic and scalable scenario generation across naturalistic driving contexts. We evaluate CrashSim against competing methods, showing that it more closely reproduces real-world pre-impact behavior, collision dynamics, collision geometry and crash-type distributions. We further use CrashSim to construct nuCrash dataset, containing over 4,000 crash and near-crash scenarios. Closed-loop evaluation of five AV planners shows that nuCrash more effectively exposes differences in planner safety capabilities than nuScenes. An LLM-assisted evaluation agent further analyzes planner failures to provide capability-level diagnoses and targeted improvement guidance. Together, CrashSim enables realistic and scalable crash generation for more informative AV safety evaluation.

Introduction

Rare crashes make real-world AV safety assessment costly, while existing generators often miss realistic pre-impact behavior and crash-type distributions. CrashSim uses retrieved real-world crash priors to guide scenario generation and supports more realistic crash scenarios and planner evaluation.

Introduction

  • Sparse crashes and severe near-crashes make real-world safety evaluation time-consuming and costly, even across millions of trips or tens of millions of driving miles.Simulation is used to evaluate rare hazardous interactions more efficiently, while realism in behavior and scene-level distributions remains important.Source: Introduction · p. 1
  • Existing methods often prioritize collision occurrence or criticality, yet may omit emergency responses, generate implausible kinematics, and overrepresent particular conflict types.These are distinct realism gaps in pre-collision interactions and scenario-set composition.Source: Introduction · pp. 1–2
  • Real-world crashes evolve through intention formation, risk awareness, emergency response, and collision outcome, with crash types occurring at markedly different frequencies.Crash data can guide both individual-scenario evolution and the distribution of scenario types.Source: Introduction · p. 3
  • Crash data are sparse, unevenly distributed, and heterogeneous, and only a limited fraction provide trajectories complete and consistent enough for direct generative modeling.Retrieval-based priors offer a way to use heterogeneous information without requiring a homogeneous crash-training corpus.Source: Introduction · p. 3
  • CrashSim retrieves contextually relevant crash priors and adapts their relative-motion patterns and behavior-stage timing to guide a diffusion-based generator on naturalistic scenes.The framework uses a small crash retrieval library rather than requiring a large crash-trajectory training dataset.Source: Introduction · p. 3
  • CrashSim more closely reproduces real-world pre-impact behavior, collision characteristics, and crash-type distributions than competing methods.It also supports nuCrash, which more clearly differentiates five planners’ safety performance than nuScenes and enables capability-level diagnoses.Source: Introduction · p. 3

Results

CrashSim reproduces real-world crash progression and crash-type distributions more faithfully than competing generators. The authors use it to build nuCrash and evaluate five planners, with stronger planner differentiation and LLM-generated capability diagnoses.

Results

Discussion

CrashSim reproduces realistic individual collision scenarios and the population-level distribution of real-world crash types. nuCrash offers a more discriminative setting for planner safety evaluation, while an LLM-based agent interprets evaluation evidence as capability assessments and improvement directions.

Discussion

  • Agreement in behavioral-stage composition, pre-impact dynamics, response timing, impact location, and crash-type distributions indicates realism at both scenario and population levels.CrashSim shifts generation from merely inducing collisions toward reproducing plausible pre-collision interaction dynamics.Source: Discussion · p. 11
  • nuCrash enriches naturalistic driving data with realistic safety-critical interactions and shows greater between-planner variation, making planner weaknesses more distinguishable.The real-world crash retrieval database can also provide empirical priors for other diffusion- or LLM-based scenario generation methods.Source: Discussion · p. 11
  • An LLM-based evaluation agent translates deterministic metrics, scenario statistics, failure patterns, and interaction contexts into capability assessments and improvement directions.The analysis supports cross-dataset reasoning about how planner capabilities change from naturalistic to safety-critical driving.Source: Discussion · p. 11
  • LLM-based diagnoses associate performance changes with behavioral patterns and scenario conditions, but attribution of underlying failure mechanisms remains relatively coarse.The authors identify deeper failure attribution as a direction for future work.Source: Discussion · p. 11

Methods

CrashSim retrieves real-world crash priors from a scene-understanding query and converts their motion and behavior stages into guidance for diffusion-based multi-agent trajectory generation. It also evaluates planners on naturalistic and generated safety-critical scenarios, using an LLM agent to diagnose capability limitations and suggest improvements.

Crash generation

Evaluation setup

  • nuCrash scenarios are generated from nuScenes scenes and retained as a fixed evaluation set shared across planners before closed-loop rollouts replace the recorded ego trajectory with planner output.Planners are evaluated on both datasets under the same closed-loop protocol using deterministic metrics and scenario-level statistics.Source: Problem formulation · p. 13, LLM-assisted planner evaluation · p. 16

Crash-prior retrieval and anchor construction

  • Retrieved crash anchors preserve relative-motion patterns and behavior-stage timings while aligning the reconstructed trajectory to the candidate vehicle's current state.CrashSim retrieves compatible interaction modes, events, and motion samples; it reconstructs the anchor from differential motion rather than copying the source crash's absolute trajectory.Source: Real-world crash-prior retrieval and… · p. 14, Diffusion-based crash scenario… · p. 15

Diffusion-based crash scenario generation

  • Diffusion guidance aligns generated trajectories with anchor behavior and stages while promoting collision, route plausibility, trajectory realism, and expected stage ordering.Guidance is applied to the candidate adversarial vehicle, while other surrounding vehicles receive realism guidance and the ego trajectory receives none.Source: Diffusion-based crash scenario… · p. 15

LLM-assisted planner evaluation

  • The LLM evaluation agent compares planner evidence across nuScenes and nuCrash to diagnose failure mechanisms, assess capabilities, and recommend targeted improvements.It derives profiles across risk anticipation, interaction reasoning, longitudinal and lateral response, comfort and stability, and mobility efficiency.Source: Methods · p. 12, LLM-assisted planner evaluation · p. 16
Loading 2610.04366v1…