PaperlayerView exact revision on arXiv

Source-linked AI summary

A Survey on Safety-Critical Driving Scenario Generation -- A Methodological Perspective

Wenhao Ding, Chejian Xu, Mansur Arief, Haohong Lin, Bo Li, Ding Zhao

arXiv:2202.02215v6cs.RO

TL;DR

Safety evaluation remains difficult because safety-critical scenarios are rare in naturalistic driving data. The survey reviews generation methods, supporting tools, and research challenges, organizing algorithms into data-driven, adversarial, and knowledge-based categories. It highlights fidelity, efficiency, diversity, transferability, and controllability as central challenges for future work.

  • Problem

    Safety-critical scenarios are rare in collected real-world driving data, limiting safety evaluation of autonomous driving systems.

  • Method

    The survey develops a methodological review that categorizes generation algorithms, summarizes simulation platforms, packages, and datasets, and examines research challenges.

  • Results

    The survey organizes existing methods into three types and identifies five challenges with corresponding potential research directions.

  • Takeaways & Limitations

    The survey provides guidance on why safety-critical scenario generation matters, how to select among existing algorithms, and which directions may improve them.

Abstract

from arXiv · show

Autonomous driving systems have witnessed a significant development during the past years thanks to the advance in machine learning-enabled sensing and decision-making algorithms. One critical challenge for their massive deployment in the real world is their safety evaluation. Most existing driving systems are still trained and evaluated on naturalistic scenarios collected from daily life or heuristically-generated adversarial ones. However, the large population of cars, in general, leads to an extremely low collision rate, indicating that the safety-critical scenarios are rare in the collected real-world data. Thus, methods to artificially generate scenarios become crucial to measure the risk and reduce the cost. In this survey, we focus on the algorithms of safety-critical scenario generation in autonomous driving. We first provide a comprehensive taxonomy of existing algorithms by dividing them into three categories: data-driven generation, adversarial generation, and knowledge-based generation. Then, we discuss useful tools for scenario generation, including simulation platforms and packages. Finally, we extend our discussion to five main challenges of current works -- fidelity, efficiency, diversity, transferability, controllability -- and research opportunities lighted up by these challenges.

I. INTRODUCTION

Safety-critical scenarios are rare in naturalistic driving, making artificial generation important for evaluating AV safety. This survey organizes generation methods, reviews supporting tools, and identifies challenges for future work.

  • Motivation: Safety-critical scenarios are extremely rare in normal driving data, despite AVs achieving over 10,000 miles without disengagement in reported normal cases.The survey uses rare emergency situations, such as a child entering the driving lane, to motivate targeted scenario generation.
  • Motivation: Artificial generation is increasingly studied by government, industry, and academia to efficiently evaluate AV safety.Relevant efforts include NHTSA test cases and simulation-framework reviews, Waymo crash reconstruction, and academic generation methods.
  • Scope: The survey reviews safety-critical scenario generation from a methodological perspective and provides a panorama of existing approaches.
  • Contributions: Existing algorithms are categorized according to the information and general framework they leverage.
  • Contributions: The survey summarizes simulators, open-source scenario-design packages, and commonly used scenario datasets.
  • Contributions: Five challenges and corresponding future directions are identified for advancing safety-critical scenario generation.

II. OVERVIEW OF SAFETY-CRITICAL SCENARIO

The survey formalizes driving scenarios through static environments, dynamic-object initialization, and sequential behaviors, then models safety-critical generation as optimizing scenario distributions against risk metrics for an AV system. It frames the optimization around representing the scenario distribution and selecting the risk metric.

  • Scenario and AV-system definitions: A driving scenario combines static environment content, dynamic-object initial conditions and properties, and their sequential behaviors.Static content includes road geometry and traffic infrastructure; dynamic objects influence the AV’s short-term decisions.
  • Scenario and AV-system definitions: The AV system comprises perception, planning, and control modules that process sensor data, produce trajectories, and execute control signals.The taxonomy figure also identifies these three AV modules as the targets of generation algorithms.
  • Safety-critical scenario generation: Safety-critical scenario generation treats the scenario distribution as parameterized and uses a metric to measure risk and safety-related properties.The metric examples include collision rate and distance driven out of the road, while the AV system maps scenarios to control-signal sequences.
  • Safety-critical scenario generation: The resulting optimization is difficult because methods must represent the scenario distribution and select an appropriate risk metric.The survey addresses these two issues in separate subsequent subsections.

B. Representation of Scenario

Scenario representation depends on the module being evaluated: perception uses high-dimensional sensing data, while planning and control use trajectories or policies. Generation must balance risk with realism, diversity, and computational uncertainty.

  • Representation of Scenario: Perception evaluation uses high-dimensional sensing data such as images and LiDAR point clouds, whereas planning and control can use trajectories or dynamic-object policies.Trajectories offer more controllability, while policy models provide greater flexibility.
  • Representation of Scenario: Directly generating high-fidelity sensing data is difficult, so differential renderers and LiDAR simulators can produce sensor outputs through ray casting.
  • Metrics for Generation: Risk metrics guide scenario optimization, but poor metrics may produce scenarios that are trivial for AVs or too rare in the physical world.The survey identifies interaction between the AV and other participants as a natural basis for risk measurement.
  • Metrics for Generation: Time-to-Collision uses relative position, speed, and leading-vehicle length to quantify collision proximity; related variants include TET and MTTC.
  • Metrics for Generation: Distance-based metrics such as PSD and deceleration-based metrics such as DRAC provide alternative ways to characterize dangerous situations.
  • Challenges: Generation must jointly ensure risk, physical realism, and broad scenario discovery rather than optimizing risk alone.The survey also highlights balancing input-modeling effort against simulation effort because uncertainty arises from both modeling and sampling.

III. DATA-DRIVEN GENERATION

Data-driven generation uses collected datasets to replay real-world scenarios or learn models for sampling new ones. Its benefits include fidelity to recorded behavior, but limited rarity and diversity motivate augmentation strategies.

  • Data-Driven Generation: Data-driven methods leverage collected datasets, either directly sampling logs or sampling from learned density models driven by random noise.
  • Data Replay: Direct dataset sampling reproduces road-test scenarios but usually suffers from the rarity of safety-critical events.
  • Data Replay: Scenario bases store company-identified important cases, while automated log-to-simulation conversion and critical-case selection support efficient testing.
  • Data Replay: Unsupervised clustering groups similar scenarios to improve testing efficiency, but whole-scenario clustering can be inefficient and inaccurate for complex scenarios.Traffic Primitives address this complexity by representing scenarios as finite building blocks.
  • Data Augmentation: Random perturbation augments scenario diversity by placing vehicle and pedestrian CAD models in simulated environments, with sensor-specific rendering such as physical-based LiDAR models.

B. Density Estimation Methods

Density-estimation methods learn scenario distributions from collected data and generate samples through probabilistic, sequential, deep generative, or imitation-learning models. These approaches trade structural modeling and naturalism against coverage outside the training distribution.

  • Density Estimation Methods: Density-estimation methods learn a model that approximates the distribution of collected driving scenarios, with categories determined by the chosen density model.
  • Bayesian Networks: Bayesian Networks represent scenario objects as nodes and their relations as edges, supporting dynamic traffic modeling, factor graphs, and importance sampling.
  • Probabilistic Models: Gaussian Processes model sequential scenario distributions, while combinations with Dirichlet Processes discover traffic primitives that can be recombined into new scenarios.
  • Deep Generative Models: Deep models can generate naturalistic sequences by conditioning an LSTM on the AV state and a high-definition map to produce surrounding-agent trajectories.The cited model is trained on normal traffic data.
  • Deep Generative Models: VAEs learn latent spaces of encounter trajectories for unseen-scenario generation, alongside other modern generative families including GANs, autoregressive, flow-based, and diffusion models.
  • Imitation Learning: Imitation learning reproduces real-world agent behavior for familiar observations, but behavior can become unreasonable and unpredictable for observations absent from the dataset.
  • Adversarial Generation: Adversarial generation actively creates risky scenarios by optimizing a generator against an AV victim model, with AV performance represented by a quantitative function Q(·, π).Because this process may focus on a small scenario set, diversity constraints or distribution entropy can be added.

A. Static Scenarios Generation

Static scenario generation attacks perception systems by optimizing rendered or point-cloud inputs, while dynamic planning and control evaluation requires sequential scenarios. Initial-condition or trajectory control reduces search complexity but can limit flexibility and diversity.

  • Static Scenarios Generation: Adversarial perception methods often optimize object poses or shapes and render the resulting scenes into camera or LiDAR inputs.
  • Static Scenarios Generation: A physical adversarial object placed on a vehicle can be ignored by LiDAR detection algorithms, and 3D printing can transfer the optimized object to the real world.
  • Static Scenarios Generation: Point-cloud attacks can directly add new points to existing data to target segmentation algorithms.
  • Dynamic Scenarios Generation: Planning and control evaluation requires dynamic sequential scenarios, which current methods divide according to scenario flexibility.
  • Initial Condition: Initial-condition methods control variables such as velocity and spawn position or provide complete trajectories, reducing search dimensionality and computational requirements.
  • Initial Condition: Gaussian parameterization can limit generated-scenario diversity, while normalizing flows and real-world-data constraints address multimodality and fidelity.

V. KNOWLEDGE-BASED GENERATION

Knowledge-based generation incorporates expert domain knowledge to make generated scenarios satisfy traffic and physical constraints. The survey covers rule-based design, learning-based guidance, predefined platforms, and constraint optimization.

  • Domain knowledge helps enforce traffic rules and physical laws that data-driven samples or adversarial examples may violate.
  • Learning-based methods condition scenario generation on expert knowledge or use that knowledge as constraints when manipulating existing scenarios.
  • Pre-defined Rules: Rule-based approaches design scenario structures and parameters directly, including random risk scenarios, logical scenarios, and natural-language traffic-scene specifications.
  • Pre-defined Platforms: Predefined scenario platforms include SMARTS, MetaDrive, CARLA-based collections, SUMMIT, and CausalCity for diverse behaviors, road layouts, mixed traffic, and causal analysis.
  • Learning-Based Methods: Knowledge can be represented through first-order logic, latent-space constraints, causal graphs, reinforcement learning, or optimization of risk-related metrics such as drivable area.
  • Generation Tools: The survey also reviews datasets, traffic simulators, and scenario-generation platforms as tools supporting knowledge-based generation.

A. Scenario Dataset

The survey compares real-world and synthetic scenario datasets across viewpoints, sensor types, annotations, and environmental diversity. These dimensions determine dataset representativeness and the scenarios or tasks they support.

  • Dataset Comparison: Table I summarizes scenario datasets across the listed views, data types, annotations, environments, and conditions.
  • Scenario datasets are collected from onboard sensors on public roads or generated in synthetic virtual worlds.
  • Real-world collection is time-consuming but more representative of real-world distributions and can support direct real-world application.
  • Collect View: Datasets differ in bird’s-eye and first-person views, with BEV supporting fixed-background behavior analysis and FPV capturing views from cameras ahead of cars.
  • Data Types and Annotations: The comparison considers RGB, stereo, LiDAR, Radar, and trajectory data, alongside 3D object, 2D object, and lane annotations.
  • Scenario Diversity: Limited diversity in weather, time, region, and traffic density produces redundant and highly safe datasets associated with the long-tail problem.

B. Traffic Simulation Platforms

Traffic simulation platforms trade realism and fidelity against computational lightness and ease of use while differing in customization, mapping, interfaces, and scenario-design support. The survey reviews simulators, editors, and reusable scenario platforms.

  • Fidelity and Efficiency: High-fidelity simulators support more realistic assessment, while lightweight simulators enable faster testing with simpler configurations.
  • Customization: Scenario customization includes vehicle positions, speeds, behaviors, traffic counts, weather, and surrounding environments.
  • Simulation Engines: Simulation engines affect vehicle-dynamics fidelity and rendered environments, with platforms built on UE4, Unity, Panda3D, or minimal back ends.
  • Maps: Maps may be real-world-based, human-designed, rule-based, or procedurally generated, supporting varied road layouts and testing settings.
  • Programming Interfaces: Python, C++, and sometimes ROS interfaces support batch evaluation and integration with open-source modules.
  • Scenario Design Platforms: Scenario tools provide reusable definitions and execution mechanisms through Python, OpenSCENARIO, domain-specific languages, XML, graphical editors, and APIs.
  • Scenario Design Platforms: SafeBench integrates eight safety-critical scenario types and four generation algorithms for systematic safety and robustness evaluation.
  • Scenario Design Platforms: CommonRoad supports reinforcement-learning motion-planner evaluation and modification, visualization, and storage of scenarios through Python and multiple specification formats.

VII. CHALLENGES AND POTENTIAL SOLUTIONS

The survey identifies fidelity, efficiency, diversity, transferability, and controllability as central challenges in safety-critical scenario generation. It argues that combining data-driven, adversarial, and knowledge-based methods may help address them.

  • Potential Solutions: The survey presents combinations of the three generation categories as promising directions for addressing these challenges.
  • Fidelity: Fidelity requires generated scenarios to remain plausible enough to occur in real traffic rather than merely being difficult.
  • Efficiency: Efficiency matters because safety-critical scenarios are extremely rare, so generation should increase the density of scenarios of interest.
  • Diversity: Diversity requires algorithms to discover and generate many different safety-critical scenarios.
  • Transferability: Transferability requires scenarios to vary across different autonomous vehicles instead of targeting only one specific vehicle.
  • Controllability: Controllability requires generative models to follow instructions or conditions so specific scenarios can be reproduced or repeated.

A. Fidelity

Safety-critical scenario generation must balance realism, computational tractability, diversity, transferability across AVs, and controllable variation. The survey highlights constraints from real-world plausibility, rare-event sampling, black-box systems, and high-dimensional sensing data.

  • Fidelity: Adversarially searched scenarios may violate basic traffic rules, making fidelity difficult when realistic constraints are hard to define.The survey suggests combining real-world data with adversarial generation as another way to address this problem.
  • Fidelity: High-dimensional sensing data requires powerful generative models, while extending visual scene-generation methods such as NeRF to large-scale traffic remains challenging.NeRF learns raycasting functions and outputs different views of a scene.
  • Efficiency: Uniform sampling is inefficient because safety-critical scenarios are rare, while importance sampling is difficult to extend to high-dimensional cases.The black-box property of victim AVs remains a major obstacle for adversarial generation.
  • Fidelity: Symbolic reasoning and causal discovery are proposed to represent scenario elements and uncover mechanisms that make scenarios safety-critical.These approaches move beyond optimization in large numerical spaces toward interpretable reasoning about risk.
  • Diversity: Current methods often search for a best safety-critical scenario while neglecting diversity, risking over-fitting when testing scenarios are too similar.The survey identifies multimodal optimization and regularization as directions for generating more varied feasible scenarios.
  • Transferability: Transferability matters because pixel-level adversarial attacks may work only on the target victim, motivating attacks at group or semantic levels.The survey treats applicability to other algorithms as a crucial factor in evaluating generation methods.
  • Controllability: Controllability supports repeating scenarios with fixed parameters or generating different scenarios under similar settings through conditional generative models.The example varies vehicle approach directions while keeping a highway environment as the shared setting.

F. Extension to Other Applications

The survey situates safety-critical scenario generation within a broader safety-evaluation agenda and considers how its methods may support other applications. It also offers combination strategies and future directions for selecting methods, incorporating knowledge, and using generated scenarios to improve robustness and generalization.

  • Extension to Other Applications: Indoor robotics applications are harder to extend because objects interact in complex three-dimensional spaces rather than mainly on a two-dimensional traffic surface.The survey specifically identifies household robots and manipulation tasks as examples requiring safety-critical scenario evaluation.
  • Survey scope: The survey categorizes methods into data-driven, adversarial, and knowledge-based generation, and reviews simulation platforms and datasets for scenario generation.It also identifies five challenges and potential directions for addressing them.
  • Extension to Other Applications: Safety-critical scenarios are needed because naturalistic and human-designed evaluations may not reveal whether AVs handle aggressive vehicles or suddenly appearing pedestrians.The survey frames artificial generation as a way to address gaps in current safety and robustness evaluation.
  • Method selection: For broad weakness discovery, combining multimodal density models with adversarial generation supports random sampling followed by safety-critical scenario search.This recommendation is presented for evaluating systems across diverse scenarios.
  • Method selection: When requirements become constraints, combining data-driven and knowledge-based generation manipulates existing scenarios but concentrates them in one cluster with low diversity.The concentration is described as useful for testing specific scenarios.
  • Method selection: Combining adversarial generation with rule-based methods can increase diversity while automatically learning scenario parameters.This combination is recommended when rules already design safety-critical scenarios.
  • Future directions: Future work should integrate inductive bias, use generated scenarios for robustness training, and investigate curriculum learning for generalization.The survey cautions that adversarial training can produce robust yet conservative systems and that selecting scenario difficulty is nontrivial.
Loading 2202.02215v6…